Spatial Computing
Spatial computing is a term used to define the various technologies allowing to deliver an immersive digital experience blending the real world with a virtual one. Although this term has been used before, it was made popular by Apple that introduced and marketed their Apple Vision Pro headset as the first Spatial Computer, which could prove as transformative as the first Personal Computer and first Mobile Computer.
Apple Vision Pro
Apple announced the Apple Vision Pro at their Worldwide Developers Conference on June 5th 2023, and released it in the US on February 2nd 2024. Apple introduced the Apple Vision Pro as “a revolutionary spatial computer that seamlessly blends digital content with the physical world, while allowing users to stay present and connected to others.” Apple Vision Pro delivers engaging entertainment experiences : “With two ultra-high-resolution displays, Apple Vision Pro can transform any space into a personal movie theater with a screen that feels 100 feet wide and an advanced Spatial Audio system. Users can watch movies and TV shows, or enjoy stunning three-dimensional movies. Apple Immersive Video offers 180-degree high-resolution recordings with Spatial Audio, and users can access an exciting lineup of immersive videos that transport them to entirely new places.”
MV-HEVC
MV-HEVC stands for Multiview High Efficiency Video Coding, and was defined in October 2014 in the Annex G of the first extension of the HEVC specification. MV-HEVC allows encoding multiple views as different layers using inter-layer predictions to increase compression efficiency by leveraging redundancies between views. At WWDC23, Apple announced that the Apple Vision Pro supports MV-HEVC for 3D movies and that bitrate savings on the second eye can reach around 30%.
SHVC, another HEVC extension standing for Scalable High Efficiency Video Coding, was concurrently defined in Annex H. SHVC shares many similarities with MV-HEVC, that were codified in Annex F, the common specifications for multi-layer extensions. SHVC was deployed starting in 2018 for ATSC 3.0 terrestrial uses cases, where the base layer is in HD and an extension layer can augment it in UHD.
Stereoscopic video
A stereoscopic video presents slightly different images to the left and the right eye to create a sense of three-dimensional depth. A stereoscopic video is typically acquired with a specific camera system that can generate two synchronous 2D videos, one per eye. The main challenge to distribute such video is to retain alignement and preserve consistent percieved video quality between eyes.
Base layer
In MV-HEVC, the base layer is the layer that can be decoded alone by an HEVC decoder for backward compatibility. It is typically the left eye, but not necessarily, such as when the capture device doesn’t have the same resolution for both views – in which case the view with the maximum resolution is typically used.
Dependent layer
In MV-HEVC, the dependent layer is the layer that is decoded using the base layer as a reference. It is typically the right eye, but not necessarily.
VPS extension
The Video Parameter Set extension is specified in the annex F of HEVC for use by MV-HEVC. It is ignored by regular HEVC decoders. Among other data, the VPS extension contains information about the number of views and their structure.
AR, XR, VR
Augmented Reality introduces additional virtual elements on the world around us, Virtual Reality replaces it entirely with imaginary worlds, and eXtended Reality can be a combination of both.
Immersive
Immersive experiences are designed to replicate sensation people would feel in the physical world. They typically includes visual, auditive and even haptic dimensions. They are meant to perceptually transport people in time and space.
Dolby Vision Profile 20
High Dynamic Range is an essential part of providing a convincing, true-to-life immersive experience. Dynamically adapting to the user lighting environment is also a specific challenge for AR/XR use cases. Dolby therefore defined a specific Dolby Vision profile for Head-Mounted Displays, called Profile 20 that Apple Vision Pro supports for 3D applications. It is based on Perceptual Quantizer. Find more details.
Head-Mounted Display (HMD)
A Head-Mounted Display, or HMD, is a device worn on the head that includes a display in front of the user eyes, usually to create virtual environments. It typically includes a Inertial Measurement Unit with accelerometers and gyroscopes, among others, to track the user head position and orientation.