VR users are currently underwhelmed by quality while operators are currently overwhelmed by the bitrates required.
Virtual Reality 360 means it doesn’t have to be this way though. One of the more puzzling aspects of the still young Virtual Reality market is that investors continue to pump in money to R&D, despite a much weaker consumer response to the technology than was initially predicted. According to Digi-Capital figures, there was a record $3bn investment in VR and AR during 2017, with half of that being raised in Q4 alone.
Undeniably, VR to date has had its issues. Initial promises of stratospheric sales were punctured by the unexpectedly high prices of the high-end kit that was meant to drive them, when it was released on the market in spring 2016. Also, consumer HMDs were inevitably tethered units that, at least to begin with, required high-powered PCs to run effectively. They also delivered a relatively low resolution experience, especially in comparison to the 4K HDR television screens that were starting to appear en masse in consumer living rooms. As a result, and perhaps remembering the hype and failure of stereo 3D a few years before, the public was skeptical.
However, that is a long way from being the end of VR’s second coming, as the continued investment shows. The premium immersive experiences that consumers demand from the technology, and that to a large extent have kept it waiting on the sidelines, are possible in a limited way in the here and now. Very soon, these experiences will be available for mainstream deployment.
The state of VR in 2018
Along with our partners at Harmonic, we’ve produced an eBook “360 Degrees of Video Opportunity, An Operator Gateway into Virtual Reality”. You can download it here.
The eBook goes into all the arguments surrounding the state of VR in 2018 in detail, looks at how operators are already monetizing VR content, examines the burgeoning field of gaming in VR, and runs through some of the early case studies of VR deployments.
It also looks in detail at some of the hurdles VR has to surmount before it transitions to the next stage and genuine mass market acceptance. Content security, more effective audio spatialization, better spec’d HMDs, device fragmentation, and more reliable, high-bandwidth broadband are all factors.
Here we’ll look at one of the most effective ways of bridging the chasm to the immersive world of VR360: tiling.
Using tiling to implement Virtual Reality 360
The key to the success of using tiling to create immersive VR360 experiences is understanding the following fact: 80% of pixels sent to the viewer are never seen. Once you begin to engineer within those boundaries, you find yourself able to produce high quality, genuinely immersive VR360 content using existing content delivery networks, while keeping latency below 20ms.
Here’s how it works.
For Tiled Streaming, the full VR panorama is first cut into rectangular tiles that are independently decodable. HEVC tends to be used here both because of its native support for Motion Constrained Tile Sets and because it currently offers the industry’s best support for resolutions of 4K and above. 4K and a move towards higher resolution displays is one of the key planks of producing genuinely immersive VR.
Tiled, encoded and packaged content is then distributed in the usual way via a CDN. Tiling only needs those tiles that are in the viewers’ Field of View to be streamed, and with current headset models only about 15% of the entire panorama is displayed. A software player tracks what part of the panorama the viewer is looking at and retrieves the individual tiles via the CDN. A base layer is also maintained, which is a low res stream of the entire panorama and means that if there is an unexpected delay in presenting the high res image, viewer continuity can be maintained.
Where bandwidth allows, adjacent tiles can also be pre-fetched, but this is not always necessary due to the way that the vestibulo-ocular reflex works. This essentially ensures the focus of the eye lags behind the movement of the head. Research by Tiled Media, who implement the system we have used in our demonstrations of a VR360 system to date, suggests that the high resolution tiles therefore don’t have to be presented until one or two frames after the head has moved. Even when running at 60fps, this give a system designed to operate with a 20ms lag plenty of spare time to swap in the new, high res tile before the eyes are fully realigned.
The result is a solution that delivers a compelling, immersive Virtual Reality 360 experience in much lower bandwidth environments. Indeed, compared to the 20Mbps that the ‘brute force’ approach of sending the entire 360 panorama requires, a viewport-adaptive system such as this can achieve the same — even superior — results in as little as 4Mbps.
Securing tiled content streams
While regular pay TV security protects premium content by ensuring it is kept in a trusted environment, with VR360 the video stream needs to be processed by a GPU. Very few devices can propose secure video environment that can include secure GPU processing. This means that the video must exist in an unscrambled form outside of the secure environment so that it can be processed.
New security technics have to be applied, based on the same principle as tiling. This security mechanism must ensure that all the high resolution tiles will be protected and that only the ones that correspond to the user’s FoV will be decrypted. Hence, even a hacker trying to exploit a security vulnerability (such as an HDMI connection) will never be able to access the full stream in high resolution. Since one user’s experience has limited value for another user, the beauty of 360° videos is that everyone is the actor of their own viewing experience and chooses the part of the content that they want to watch.
Protecting tiled content in a VR360 ecosystem is therefore a priority for any successful commercial deployment.
Virtual Reality 360: The future is now
There are other advances that are pushing VR technology forward at a rapid pace. High Order Ambisonic audio is providing increasingly accurate 3D audio at low computational cost; standards bodies are becoming increasingly active in the arena; and all that billions of dollars of investment is ensuring the HMDs are improving rapidly.
For the consumer, this will be the most obvious change. Tethered units will quickly become things of the past, while field of view is expanding all the time and native resolutions of 4K per eye are well within scope of future generations of equipment. This means that the long-term prospects of VR are as good as they have ever been. Think of 2016 and 2017 as a false start rather than as uncovering fundamental flaws in the technology.
As our eBook says as part of its conclusion, “VR360 content changes the viewer’s role from passive spectator to active participant. We believe Virtual Reality 360 is the beginning of something big and that operators that have participated in the VR360 adventure will be better equipped to respond to future generations of active viewers.”
