The inside story of our quest for visual realism in VR

The Challenge

The challenge DSR faces as they pursue their quest for visual realism is easily summed up: The technology needed to pass the visual Turing test, especially in a consumer headset, doesn’t yet exist. While Quest and Quest 2 create compelling 3D visual experiences, they can’t yet compete with our experiences in the real world. The obvious current limitation is resolution, but the challenges run far deeper. VR introduces a slew of new issues that simply don’t exist with today’s 2D displays, including vergence-accommodation conflict, chromatic aberration, ocular parallax, and pupil swim. As a result, there are many obstacles to be overcome, a great deal of research to be done, and a lot of user studies to be conducted before we can get close to a fully realistic VR visual experience. The innovations needed to close the gap fall into several major categories.

For starters, resolution is an issue. The problem is that VR headsets have much wider fields of view than even the widest monitor, so whatever pixels are available have to be applied across a much larger area than for a 2D display, resulting in lower resolution for a given number of pixels. For example, 20/20 vision across the full human field of view would require about 13,000 pixels horizontally — far more than any existing consumer display. (The reality isn’t quite that bad, since the eye doesn’t have the ability to perceive high resolution across the full field of view, but the magnitude of the challenge still applies.) And not only are a lot more pixels required, but the quality of those pixels needs to increase. Today’s VR headsets have substantially lower brightness and contrast than laptops, TVs, and mobile phones. As such, VR can’t yet reach the level of fine detail and accurate representation that we’ve become accustomed to with our 2D displays.

Additionally, the lenses used in current VR displays often distort the virtual image, reducing realism unless the distortion is  fully corrected in software — which is challenging because the distortion varies as the eye moves to look in different directions. Moreover, while it’s not part of realism, headsets can be hard to use for extended periods of time because that distortion, as well as the headset’s weight, can cause temporary discomfort and fatigue. And there’s one more key element, which could be considered part of resolution but is so crucial that it belongs in its own category: the ability to focus properly at any distance. We’ll explain that last point and dive into it shortly, because it’s at the heart of our story today.

In order to fully address the above gaps, Zuckerberg and Lanman believe that passing the visual Turing test will require building a new tech stack that includes:

  • “Varifocal” technology that provides correct depth of focus (versus a single fixed focus), thereby enabling clearer and more comfortable vision within arm’s length for extended periods of time
  • Resolution that approaches and ultimately exceeds 20/20 human vision
  • Distortion correction to help address optical aberrations, like color fringes around objects and image warping, that can be introduced by viewing optics
  • And high dynamic range (HDR) technology that expands the range of color, brightness, and contrast you can experience in VR

Developing all these capabilities is necessary (and hard!) but not sufficient. All of that ultimately needs to fit into a more comfortable headset suitable for consumer use, and that means DSR has to not only advance the state of the art on multiple display axes but also build complete display systems well beyond what exists today — and that takes the challenge to another level. But it’s a challenge that DSR is taking on — and a challenge that Zuckerberg believes is essential to solve to get to the next generation of VR.

Lanman notes the complexity of the task: “Designing and building headsets that incorporate that collection of technologies is difficult and time-consuming work because with headset displays, all technical systems are interconnected. Everything competes for that same size, weight, power, and cost budget, while also needing to fit into a compact, wearable form factor.” And it’s not just a matter of squeezing all the technology into a tight budget — each element of the stack must also be compatible with all the others. For example, certain eye tracking technologies must be paired with specific types of display lenses in order to function properly.

DSR has tackled this head-on with an extensive series of prototyping efforts, ranging from individual technologies to full systems, that map out and push the boundaries of the vast VR display design space, followed by user studies run on those prototypes to assess progress toward passing the visual Turing test. The tangible result of this is on display at RL Research in Redmond: an entire wall of prototypes that collectively explore a wide spectrum of technology for next-generation VR displays — a living history of DSR’s quest for visual realism.

Leave a Comment