Advanced design and production tools that enable broadcasters to create virtual objects that appear as if they’re actually in the studio have now been available for a few years, but have only become a mainstream feature for major TV networks over the last year.
Broadcast is increasingly adopting AR graphics for enhanced storytelling, allowing for better interaction between presenters and graphics objects and even remote locations. AR gives broadcasters an extra level of freedom to tell a complex story in a very visual way with the presenter driving the narrative. Augmented Reality applications allow for visually engaging representations of data which can then be better explained by the presenters placed on the set.
“Augmented Reality allows for displaying data-driven graphics along with real images, where real footage or live videos are mixed with virtual backgrounds or scenes, chroma keyed talents and data-driven 3D graphics, which can interact with the talents creating an attractive environment for the audience. This “mixed reality” allows for creating virtual environments where visually engaging representations of the data can be better explained by the presenters, making complex data easier to understand while enhancing the storytelling,” says Miguel Churruca, Marketing and Communications Director for Brainstorm.
Having had a few years of experimentation behind them, broadcasters now have a much better idea of where AR makes sense, how to use it, and what kind of AR elements make sense. And we’re now seeing more realistic graphics, with the introduction of hyper-realism. Using render engines broadcasters are bridging the gaps and removing barriers to entry.
Above: BBC was one of many broadcasters to use AR graphics at World Cup 2018
Especially in areas like live sports production, 2018 has been a breakthrough for AR graphics adoption thanks to big ticket events like the Winter Olympics and the World Cup.
AR Market: Hitting the Mainstream
The market for AR is still in the front end of the adoption curve. When it comes to the mainstream use of AR graphics in broadcast, if we assume a 100% adoption curve would mean that every local market TV station is using AR to the same extent that they deploy regular graphics such as graphics and lower thirds, then over the last year or two, the industry has been making rapid progress to reach a adoption levels of 30-40%.
“I think it’s a little soon to expect mass adoption of AR/VR solutions, but we are seeing progress. The availability of flexible and reliable optical tracking systems, together with the recent publicity around the use of the Unreal engine in AR/VR applications, is beginning to have a real impact,” says Olivier Cohen, Senior Product Manager at ChyronHego.
Broadcasters continue to refine the AR viewing experience: “Apart from the quality of the graphics and backgrounds, the most important challenge is the integration and continuity of the whole scene,” says Churruca. “Having tracked cameras, remote locations and graphics moving accordingly, perfect integration, perspective matching and full broadcast continuity are paramount to provide the audience with a perfect viewing experience of AR graphics.”
The growing maturity of the industry has helped graphics vendors develop better solutions that are improving the quality of the augmented reality viewing experience. Brainstorm’s Churruca elaborates on what details help to improve the viewer experience: “AR is a team job, involving design, data gathering and management, virtual sets, camera tracking and much more. So, the perfect integration between the different real and virtual objects and the backgrounds becomes essential, but what really makes the difference for the audience is to be unable to tell whether the images they are watching are real videos or digital renders. However, for virtual set production and live broadcast operation, photorealism is a complicated challenge because of the constraints of real-time rendering and operation.”
Above: Al Arabiya used AR graphics for the 2016 US elections
These technologies offer really exciting new opportunities for revenue generation. At a basic level, using AR and virtual set tools in the studio to augment storytelling lowers costs, enables fast changes to any environment.
Stations can use virtual set technology to create a lot more content and greater numbers of shows within the same studio space. Without the time and hassle of building and dismantling physical sets between shows, new revenues can open up thanks to more efficient of valuable real estate in the studio.
Plus the AR graphics can serve as virtual advertisements, together with sponsored virtual graphics — both of which offer new revenue sources. For example, a station could sell sponsorships of virtual players or data and stats; “this player segment is brought to you by (the sponsor),” along with a huge 3-D rendition of the sponsor’s logo.
As the technology matures and becomes more widely adopted, there is the potential to monetize Augmented Reality content through programmatic ads inserted into a virtual environment, based on information about a specific user such as their location or preferences, or even allowing the user to control the environment through their device.
A snapshot of the industry’s leading graphics solutions offerring Augmented Reality.
Vizrt: Viz Virtual Studio with Viz Engine
Viz Virtual Studio gives producers the ability to tell stories easily without worrying about the limitations of physical studio facilities. AR solution utilizes the same tool that Vizrt uses to render all their graphics and playout video: Viz Engine for rendering, and adds on the Viz Virtual Studio software on top of that to manage the tracking data coming in for any tracking system.
This means that the same engine that is being used for a video wall for one show, can be used to display AR graphics for the next, giving the broadcaster a lot of flexibility and saving them money. Viz Virtual Studio and Viz Engine can be used with any tracking system so the customer is free to decide which works best for their budget.
- Create complex, interactive 3D virtual sets and immersive graphics using an intuitive interface
- Allows a single operator to handle complex productions that would normally require a team of operators
- Frees up staff concentrate on other tasks, improving overall efficiency.
Brainstorm: eStudio and InfinitySet
Brainstorm Multimedia’s augmented reality solutions offer a range of tools within its eStudio and InfinitySet products. eStudio serves as the render engine on which all other Brainstorm products run. InfinitySet supports everything from simple and complex virtual sets.
“AR is fully implemented in all the Virtual Set range of Brainstorm, headed by eStudio and InfinitySet, but is also available using our template-based solutions like SmartSet. The requirements can be as simple as a single workstation with single or multiple cameras, tracked or fixed, a chroma set, etc., and the systems are modular enough to match any requirements, so the installation can be as complex as required,” said Brainstorm’s Miguel Churruca.
- Compatible with fixed and tracked cameras, external feeds with embedded tracking data.
- Advanced compositing features, realistic real-time 3D scenes.
- Allows for combined Chroma Keying, even within the same production.
Ross Video: XPression
XPression is Ross Video’s line of real-time motion graphics systems, clip servers, workflow tools and software applications that powers its augmented reality offerings that let stations extend the story telling tool set with floating augmented reality elements, and customizable virtual sets:
“Our AR solution is comprised of multiple parts, and can be customized to meet a customer’s needs and requirements. We offer an end-to-end solution that includes robotic cameras, graphic engines, a control interface, and delay cards or any combination of parts. We work with most tracking systems in the industry, so if a customer does not wish to use our robotic camera system, our offering will adapt,” said Ross Video’s Gideon Ferber.
- Utilizes “blended” environments that eliminate the need for video walls and on-set monitors.
- Options for income sources through sponsorship of specific virtual elements.
- Works with a variety of robotic cameras, production switchers, motion graphics and automated production control systems.
Avid provides a full solution for virtual studios, camera tracking, augmented reality and video wall control enabling the broadcaster to work within a single unified workflow. RealSet features a variety of capabilities for creating 3D virtual objects that look like part of the actual environment:
It can also be used with Avid’s other tracking solutions, including Pattern Recognition system and opto-mechanical sensors, as well as third-party tracking systems.
“With a full set of augmented reality tools it’s easier than ever to change the look and feel of your studio environment and create compelling virtual set piece. It provides a feature-rich yet cost-effective virtual studio and augmented reality solution for any production need and budget,” says Avid’s Haim Halperin.
- Can be used for wide variety of indoor/outdoor productions.
- 3D objects remain tied to their exact position — regardless of camera motion.
- No chroma keying is required; inserted graphics are introduced as a foreground layer.
Virtual studios typically broadcast from a green screen set. AR comes into play where there is a physically-built set in the foreground, and augmented graphics and props are placed behind the camera. Some scenarios could have no physical props with everything behind and in front of the presenter being graphics.
AR graphics and virtual sets require all of the hardware and software in the studio, along with AR camera operators and graphics administrators to control what’s being shown on virtual screens built into the sets.
The race is on to perfect virtual environments and AR production solutions. Much of the research is centered on games engines, like the Unreal Engine from Epic Games or Unity - which were originally designed to quickly render polygons, textures and lighting in video games.
Graphics packages tied to game engines give you better graphics, animation, physics, you can really create engaging and high-quality content. Using game engine technology to achieve high quality rendering in realtime means that, once something is shot on set, there will be no post production.
Games engines create a level of realism unrivalled by any broadcast character generator but they need to be reprogrammed first so that the games software matches camera-acquired signals like timecode. Graphics solution providers like Ross Video and Vizrt are currently integrating the Unreal Engine in their products.
The Future Group for example, worked with Ross Video to rewrite the Unreal code to comply with genlock. Through this partnership with The Future Group, Ross Video created a solution called Frontier aimed at local broadcasters that increases quality of virtual sets and AR, harnessing the power of the rendering engine in XPression.
Above: A virtual set created with Frontier
Unreal Engine is on its way to becoming a true game changer in virtual production and augmented reality graphics.
Middle East Case Study
Al Arabiya leverages AR, virtual extensions in studio for World Cup
Al Arabiya used augmented reality and virtual set extensions for its World Cup 2018 coverage. The Arabic language network, under the creation direction of Fadi Rani, combined visual storytelling technology with its existing studio in Dubai.
The studio was been outfitted with a corner green screen — and the network is using five Vizrt Viz Engines to insert virtual set extension. These designs, which were created with Viz Virtual Studio, feature curved walls and a faux video wall that is often fed graphics matching the network’s World Cup open. Along with the extensions, many of the augmented reality graphics are inserted into the set throughout the coverage.
These elements, such as flag and ranking data, carry through the droplet-like “particles” from the opening sequence for Al Arabiya’s coverage. In addition, Vizrt technology was used to insert a series of concentric rings into the studio ceiling floating above the main anchor desk. These elements complement the set’s real header element.
In addition to making the set look more expansive, the augmented rings also serve a practical purpose — as a sort of “portal” that additional info graphics can descend from. The rings can also be used for a ring of “hanging” flags showcasing the many nations that make up the World Cup.
Additional augmented reality elements included a floating field along with jerseys representing players and their positions. Key people, including players and coaches could also be inserted into the studio on giant floating panels.
In addition, similar floating panels were used to insert video “screens” to showcase game and topical footage. Like many networks, Al Arabiya made use of 3D augmented reality models of the various gameplay stadiums along with virtual maps spotlighting the locales.
In addition to the spherical dot texture borrowed from the network’s World Cup graphics, other elements feature a hexagon pattern overlay hinting at the familiar black and white pattern of European footballs. The camera tracking was powered by stYpe’s RedSpy and stYpekit on a crane.