Technically Interesting New Papers At Siggraph 2017

Both Oculus (Facebook) and Microsoft’s are presenting interesting technical research  papers at Siggraph 2017 (July 30th to August 3rd) that deal with Vergence/Accommodation (VAC).  Both have web pages (Oculus link and Microsoft link) with links to relatively easy to follow videos and the papers. But readers should take to heed the words on the Microsoft Page (which I think is applicable to both): “Note that this Microsoft Research publication is not necessarily indicative of any Microsoft product roadmap, but relates to basic research around holographic displays.” I can’t hope to try and get into all the technical details here, but they both have a lot well explained information with figures and for those that are interested, you can still learn a lot from them even if you have to skip over some of the heavy duty math. One other interesting thing is that both Oculus and Microsoft used phase controlled LCOS microdisplays at the heart of their technologies.

Briefly, VAC is the problem with stereoscopic 3-D where the apparent focus of objects does not agree with were they seem to appear with binocular vision. This problem can cause visual discomfort and headaches. This year I have been talking a lot about VAC thanks first to Magic Leap (ML article) and more recently Avegant (Avegant VAC article ) making big deals about and both raising a lot of money (Magic Leap over $1B) as a result. But least you think Magic Leap and Avegant are the only ones, there are dozens of research groups over the last decade working on VAC. Included in that number is Nvidia with a light field approach that they presented a paper in 2013 also at Siggraph (The 2013 Nvidia Paper with links embedded at the bottom of the Abstract to more information and a video)

The Oculus paper has a wealth of background/education information about VAC and figures that help explain the concepts. In many ways it is a great tutorial. They also have a very lengthy set of references that among other things confirm how many different groups have worked on VAC and this is only a partial list. I also recommend papers and videos on VAC by Gordon Wetzstein of Stanford. There is so much activity that I put “Everywhere” in the title.

I particularly liked Oculus’s Fig. 2 which is copied at the top of this article (they have several other very good figures as well as their video). They show the major classes of VAC, from a) do nothing, b) change focus (perhaps based on eye tracking), to c) Multifocal which is what I think Magic Leap and Avegant are doing, to d)&e) Oculus’s “focal surfaces(s), to f) light fields (ex. Nvidia’s 2013 paper). But light fields are in a way a short cut compared to real/true holograms which is what Microsoft’s 2017 paper is addressing (not shown in the table above but discussed in Oculus’s paper and video).

I put the “real” in front of the work “hologram” because confusingly Microsoft, for what appears to be marketing purposes, has chosen to call stereoscopic merged reality objects “holograms” which scientifically they are not. Thanks to Microsoft’s marketing clout and others choosing “if you can’t beat them joint them” in using the term, we now have the problem of what to call real/true holograms as discussed in Microsoft’s 2017 Siggraph paper.

High Level Conceptually:

• Light Fields are a way to realize many of the effects of holograms such such as VAC and being able to see around objects. But light fields have piece-wise discontinuities. They can only reduce the discontinuities by massively trading off resolution; thus they need massive amounts of processing and native display resolution for a given visual resolution. Most of the processing and display resolution never makes it do the eye as based on where the eye is looking and focused, all but a small part of the generated image information is never seen. The redundancy with light fields tends to grow with a square law (X and Y).
• Focus planes in effect try and cut down the Light Field square law redundancy problem by having the image redundancy grow linearly. They need multiple planes and then rely on your eye to do the blending between planes. Still the individual planes on “flat” and with a large continuous surface there would be discontinuities at the point where it would have to change planes (imagine a road going off in the distance).
• Oculus Surfaces are in essence and improvement on focus planes where the surfaces try to conform more to the depth in the image and reduce the discontinuities. One could then argue whether it would be better to have more simple focus planes or fewer Focus Surfaces.
• Holograms have at least an “n-cube” problem as they conceptually capture/display the image in X, Y, and Z. As the resolution increases the complexity grows extremely fast. Light fields have sometimes been described as “Quantized Holograms” at they put a finite limit on the computational and image content growth.

Oculus’s Focus Surface Approach

In a nutshell, Oculus is using an eMagin OLED to generate the image and a Jasper Display Phase Shift LCOS device to generate a “focus surface”. The focus changes focus continuously-gradually, and not on a per-pixel basis, which is why they call is a “surface”.  The figure on the right (taken from their video) shows the basic concept of a “focus surface” and how the surface roughly tracks the image depth. The paper (and video) go on to  discuss how having more than one surface and how the distance approximation “error” would compare with multi-focus planes (such as Magic Leap and Avegant).

While the hardware diagram above would suggest something that would fit in a headset, it is still at the optical breadboard stage. Even using microdisplays, it is a lot to put on a person’s head. Not to mention the cost of having in effect two displays (the LCOS one controlling the focus surface) plus all the additional optics. Below is a picture of the optical breadboard.

Microsoft (True This Time) Holograms

While Oculus’s hardware looks like something that could fit in a headset someday, Microsoft is much more of a research concept, although they did show a compact AR Prototype “glasses” (shown at right) that had a small subset of the capability of the larger optical breadboard.

Microsoft’s optical breadboard setup could support either Wide FOV or Multi-Focal (VAC) but not both at the same time (see picture below). Like other real time hologram approaches (and used by Oculus in their focal surface approach), Microsoft uses a Phase LCOS device.The Microsoft paper goes into some of the interesting things that can be done with holograms including correcting for aberrations in the optics and/or a person’s vision.

In many ways Holograms ultimate end game in display technology where comparatively everything else in with VAC is a hack/shortcut/simplification to avoid the massive computations and hardware complexities/difficulties of implementing real time holograms.

Resolution/Image Quality – Not So Much

The image quality in the Oculus Surface paper is by their admission very low both in terms of resolution and contrast. As they freely admit, it is a research prototype and not meant to be a product.

Some of these limitations are the nature of making a one-off experiment as the article points out but some of the issues may be more fundamental physics. One thing that concerns me (and pointed out in the article) in the Oculus design is that they have to pass all three colors through the same LC material and the LC’s behavior varies with wavelength. These problems would become more significant as resolution increases. I will give the Oculus paper props for both for is level of information and candor about many of the issues; it really is a very well done paper if you are interested in this subject.

It is harder to get at the resolution and image quality aspects of the the Microsoft Hologram paper as they show little images from different configurations. They can sort of move the problems around with Holograms; they can tune them and even the physical configuration for image quality, pupil size, or depth accommodation, but not all at the same time. Digital/real-time holograms can do some rather amazing things as as the Microsoft paper demonstrates but but they are still inordinately expensive both to compute and display and the image quality is inferior to more conventional methods. Solving for image quality (resolution/contrast), pupil/eyebox size, and VAC/image depth simultaneously makes the problems/cost tend to take off exponentially.

Don’t Expect to See These In Stores for Decades, If Ever

One has to realize that these are research projects going for some kind of bragging rights in showing the technical prowess, which both Oculus and Microsoft do impressively in their own ways. Note the Nvidia Light Field paper was presented at Siggraph 2013 years ago and supporting decent resolution with Light Fields is still a very far off dream. If their companies thought these concepts were even remotely practical and only a few years away, the companies would have kept them deep dark secrets. These are likely seen by their companies as so out in the future that there is no threat to letting their competition see what they are doing.

The Oculus Surface approach is conceptually better on a “per plane” than the “focus planes” VAC approaches, but then you have to ask are more simple planes better overall and/or less expensive? At a practical level I think the Oculus Surface would be more expensive and I would expect the image quality to be considerably worse. At best, the Oculus Surface would be a stop-gap improvement.

Real time high resolution holograms that will compete on image quality would seem to be even further out in time. This is why there are so many companies/researchers looking at short cuts to VAC with things like focus planes.

VAC in Context – Nowhere

VAC has been a known issue for a long time with companies and researchers working in head mounted displays. Magic Leap’s $1+B funding and their talk about VAC made it a cause célèbre in AR/VR and appears to have caused a number of projects to come out from behind closed doors (for V.C. funding or just bragging rights).

Yes, VAC is a real issue/problem particularly/only when 3-D stereoscopic objects appear to be closer than about 2 meters (6 feet) away. It causes not only perceptual problems, but can cause headaches and make people sick. Thus you have companies and researchers looking for solutions.

The problem IMO is that VAC is would be say about 20th (to pick a number) on my list of serious problems facing AR/VR. Much higher on the list are based image quality, ergonomic (weight distribution), power, and computing problems. Every VAC solution comes at some expense in terms of image quality (resolution/contrast/chromatic-abberations/etc).

Fundamentally, if you eye can pick what it focuses on, then there has to be a lot of redundant information presented to the eye that it will discard (not notice) as it focuses on what it does see. This translates into image information that must be displayed (but not seen), processing computations that are thrown away, and electrical power being consumed for image content that is not used.

I’m Conflicted

So I am conflicted. As a technologist, I find the work in VAC and beyond (Holograms address much more than VAC) fascinating. Both the Oculus and Microsoft articles are interesting and can be largely understood by someone without a PhD in the subject.

But in the end I am much more interested in technology that can reach a sizable market and on that score I don’t understand all the fuss about VAC.  I guess we will have to wait and see if Magic Leap changes the world or is another Segway or worse Theranos; you might be able to tell which way I am leaning based on what I understand.

Today, the image quality of headsets is pretty poor when compared to say direct view TVs and Monitors, the angular resolution (particularly of VR) is poor, the ergonomics are for the most part abysmal, and if you are going to wireless, the batteries are both too heavy and have too short a life. Anything that is done to address VAC makes these more basic problems not just a little worse, but much worse.