Meta’s display chief names 10 features for a perfect VR headset

Image: Meta

There are still many obstacles to overcome before virtual reality reaches its technical potential. The head of Meta’s display division lists the challenges.

Douglas Lanman has been with Meta for eight years and heads the department that develops visualization systems. In June, his team unveiled new prototypes of VR glasses. The prototypes face several technical challenges such as resolution, brightness and form factor. The researchers’ goal is a VR display capable of visually reproducing physical reality while it fits into a compact headphone.

At this year’s Siggraph conference, Lanman gave a presentation on the the top ten obstacles yet to be overcome in the development of a near-perfect VR headset. The following is a list of these ten challenges, including a brief explanation. You can find the video of Lanman’s speech at the end of the article.

The ten challenges

Higher resolution

Current VR headsets don’t come close to human vision in terms of resolution. In order for virtual worlds to appear real and sharp as if they were physical and for texts to be read well even at medium distances, the resolution of VR displays must increase significantly.

Metastates 8K per eye it’s a pixel density of 60 PPD as a preliminary goal. For comparison: Meta Quest 2 (review) only achieves 2K per eye and 20 PPDs.

The higher the resolution, the sharper the image will be. Oculus Rift, Meta Quest 2 and the Butterscotch prototype in comparison. | Image: Meta

With the Butterscotch prototype, Meta has developed an experimental headset that allows researchers to experiment with “retinal resolution” and evaluate its immersive effect. The development and production of high resolution displays is not the biggest problem. Rather, the question is to house the computing power to power high-resolution displays. Foveated rendering and cloud streaming might help, but they present great technical challenges in their own right.

Larger field of view

Much needs to be done in terms of the field of view as well, notes Lanman. T.The human horizontal field of view is approximately 200 degrees wide. Commercially available VR headsets generally achieve a horizontal field of view of 100 degrees. But there is still room for improvement in the vertical field of view as well.

A wider field of view arises great challenges for lens technology, which is manifested by distortions of the image at the edges of the visual field. Here too the question of computing power arises. The wider the field of view, the more pixels the VR headset must be displayed. This results in higher power requirements and more waste heat.

Ergonomics

VR headsets are still heavy and bulky. Meta Quest 2, for example, weighs more than half a pound and protrudes nearly 3 inches from the face. VR headphones should ideally be comfortable to wear for longer periods of time and be much tighter and lighter in design.

The image shows the Holocake-1 prototype, which comes in the form of sunglasses. A 2016 Oculus Rift is transparently shown Image: Meta

Pancake lenses and holographic lenses could help. Meta’s fully functional Holocacake 2 prototype shows the direction the form factor might take. The problem: Holocake 2 uses custom lasers as a light source, which they are not yet developed to the point of being mass-produced.

Vision corrected display

The perfect VR headset should be able to detect and compensate for visual deficiencies in users so that conventional glasses or contact lenses are not needed to see well in virtual reality. Will the optical frames fit the VR headset without scratching the lenses or pushing the wearer’s face down? Consumers shouldn’t have to deal with such questions in the future.

The problem could be solved with special attacks or, even better, with a lens that can be adapted to the user’s visual acuity.

There are already attachments for lenses with vision correction. But they are not an ideal solution. | Image: Meta

Variable focus

The human eye cannot focus naturally in VR environments, which is particularly annoying at close range and can lead to eye strain and headache after a longer period of time. This phenomenon is referred to in technical jargon as a vergence-accommodation conflict.

To solve this problem, Meta researchers have developed a display that supports “progressive vision. ” The display simulates different focal planes and blurs and helps the eye see the virtual world as if it were originally from nature. Meta’s varifocal prototypes are called “Half-Dome”.

Eye tracking for everyone

Eye tracking is a key technology in virtual reality. It is the foundation of many other important VR technologies such as progressive vision (see point 5), foveate rendering and distortion correction (see point 7). It also allows for eye contact in social experiences and new forms of interaction.

The shape of the pupil varies from person to person. This is a challenge for eye-tracking systems. | Image: Meta

The problem with eye tracking is that it does it doesn’t work equally well for all people and occasionally has dropouts. A reliable solution with a broad demographic coverage is needed. Otherwise, the technology frustrates instead of helping and is rejected by consumers.

Distortion correction

Lenses inherently introduce image distortion that must be corrected by software. The slightest movement of the pupil causes subtle but noticeable distortions. They prevent visual realism, especially in combination with other technologies such as progressive displays.

To speed up the development of correction algorithms, Meta researchers developed a distortion simulator. It can be used to test different objectives, resolutions and fields of view without having to build test headphones and special objectives.

High dynamic range (HDR)

Physical objects and environments are much brighter than VR displays, even indoors with artificial lighting. Meta has made Starburst, a VR prototype capable of displaying up to 20,000 nit. For comparison, a good HDR TV offers several thousand nits while Meta Quest 2 only offers 100 nits.

Mark Zuckerberg holds the Starburst prototype. | Image: Meta

Starburst can realistically simulate lighting conditions in closed rooms and night environments. The current prototype is so heavy that it hangs from the ceiling and is extremely energy hungry.

According to Meta, HDR contributes more to visual realism than, say, variable resolution and focus, but it’s the furthest away from practical implementation.

Visual realism

The perfect VR headset must be translucent in both directions. VR users should be able to see the environment as much as the environment can see VR users. This is for reasons of user comfort and social acceptance.

The sensors of the headphones record the environment and display it as a video image in virtual reality. This technology, called passthrough, already exists in commercially available VR headsets, albeit of rather poor quality. Meta Quest 2, for example, offers a grainy black and white passthrough mode. Meta Quest Pro should greatly improve this display mode with higher resolution and color.

On the left is an Oculus Rift that blocks the VR user’s eye view. On the right is a first reverse-passthrough prototype. | Image: Meta

However, this does not yet reach a perfect reconstruction of the physical environment. One of the remaining problems is that passthrough technology captures a perspective of the world that is spatially displaced by the eyes, which can be irritating during prolonged use. To this end, Meta is looking for a synthesis of the gaze assisted by artificial intelligence that generates perspective points of view corrected in real time and with high visual fidelity.

In the reverse case, which Meta calls “Reverse step”, Strangers see the eyes and faces of VR users and can then make eye contact or read facial expressions. This could be made possible by sensors facing inwards and an external display (light field).. This technology still is far from ready for the marketas Metas research shows.

Facial reconstruction

Co-presence and metaverse telephony are the ultimate goals of Meta. The company wants people to one day meet in virtual spaces and feel like they’re present in the same room. To that end, Meta is researching photorealistic codec avatars, but they are still very expensive to produce and calculate.

A first step in this direction are VR headsets that can read the facial expressions of VR users in real time and transfer them to virtual reality. Quest Pro will be the first Meta headset to offer face recognition.

Quest Pro is able to recognize facial expressions and transfer them to VR. | Bild: Meta

Speech on the Siggraph by Douglas Lanman

Note: Links to online shops in articles can be so-called affiliate links. If you buy via this link, MIXED receives a commission from the provider. For you the price does not change.