So what is Magic Leap doing?  That is the $1.4 billion dollar question. I have been studying their patents as well as videos and articles about them and frankly a lot of it does not add up.   The “Hype Factor” is clearly off the chart with major and high tech news/video outlets covering them with a majors marketing machine spending part of the $1.4B, yet no device has been shown publicly, only a few “through the Magic Leap” online videos (6 months ago and 1 year ago).   Usually something this much over-hyped ends up like Segway (I’m not the first to make the Segway comparison to Magic Leap) or more recently Google Glass.

Magic Leap appears to be moving on many different technological fronts at once (high resolution fiber scanning display technology, multi-focus- combiner/light fields, and mega-processing to support the image processing required) which almost always is a losing strategy even for a large company no less a startup, albeit a well funded one. What’s more, and the primary subject of this article, they appear to be moving on many different fronts/technologies with respect to the multi-focus-combiner.

Based on the image above from Wired in April 2016 and other articles talking about a “photonic chip,” a marketing name for their combiner not used in any of their patent applications that I could find.   By definition, a photonic device would have some optical property that is altered electronically, but based on other comments made by Magic Leap and looking at the patents, the so called “chip” is just as likely a totally passive device.

It is also well known that Magic Leap is working on piezo scanned laser fiber displays, a display technology initially developed by Magic Leap’s CTO while at the University of Washington (click left for a bigger image). Note that is projects a spiraling cone of light.

A single scanning display is relatively low resolution and so to achieve Magic Leaps resolution goals will require arrays of these scanning fibers as outlined in their US Application 2015/0268415.

Magic Leap is moving in so many different directions at the same time. I plan on covering the scanning fiber display in more detail much more detail in the near future.   

Background – Nvidia and Stanford Light Fields

A key concept running through everything about Magic Leap it is that their combiner supports at least multiple focus depths at the same time.   The term “Light Fields” is often used in connection with Magic Leap, but what they are doing is not classic light fields such as Nvidia has demonstrated (very good article and video is here).   Or even what Stanford’s Gordon Wetzstein work talks about with compressive light field displays (example here) and several of his YouTube videos, in particular this one that discusses light fields and the compressive display.   (More on this background at the end). 

A key think to understand about “light fields” and Magic Leaps multi-focus-planes is that they are based on controlling the angles of the rays of light as it controls the focus distance.   The rays of light that will make it through the eye’s pupil from a point on far away objects come in nearly parallel, whereas the rays from a nearby point have a wider range of angles.

Magic Leap Patents

Magic Leaps patents show a mix of related and very different types of waveguide combiners.   Most in-line with what Magic Leap talks about in the press and videos are the ones that include multi-plane waveguides and scanned laser fiber displays.   These include US patent applications US20150241705 (‘705) and the 490 page US20160026253 (‘253).  I have clipped out some of the key figures from each below (click on the images to see larger images).

Fig. 8 from the ‘705 patent uses a multi-layer electrically switched diffraction grating waveguide (but they don’t say what technology they expect to use to cause the switching). In addition to switching each diffraction grating makes the image focus differently as shown in Fig. 9.  While this “fits” with the “photonic chip” language by Magic Leap, I’m less inclined to believe this is what Magic Leap is doing based on the evidence to date (although Digilens has developed switchable SBGs in their waveguides).

Fig. 6 likely comes closer to what Magic Leap seems to be working on, at least in the long term. In this case there is one or more laser scanning fiber displays for each layer of the diffraction grating (similar to Fig. 8 but passive/fixed). The gratings layers in this setup are passive and based on which display is “on” chooses the grating layer and thus chooses the focus.  Also note the ” collimation element 6” between the scanning fibers 602a-e and the waveguide 122. They take the cone of rays from the spiral scanning fiber and turns them into an array of parallel (collimated) rays. Below shows a prototype from the June 2016 “Wired” article with two each of red, green, blue fibers per eye (6 total)which would support two simultaneous focus points (in future articles I plan on going into more about the scanning fiber displays).

Above I have put together a series of figures from Magic Leap’s US patent application 2015/0346495.  Most of these are difference approaches to accomplish essentially the same effect, namely to create 2 or more images in layers that appear to be in focus at different distances.  In some approaches they will generate the various focused images time sequentially and rely on the eye’s persistence of vision to fuse them (the Stanford Compressive Display works sequentially).  You may note that some of the combiner technologies shown above are not that flat including what is known as  “free form optics” (Fig. 22G above) that would be compatible with a panel (DLP, LCOS, or Micro-OLED display).

And Now for something completely different

To the left patent application 2015/0346495 that shows a very different optical arrangement with a totally different set of inventors from the prior patents.   This device supports multiple focus effects via a Variable Focus Element (VFE).   What they do is generate a series of images sequentially and change the focus between images and use the persistence of the human visual system to fuse the various focused images.

This is a totally different approach to achieve the same effect.   It does requires a very fast image generating device which would tend to favor DLP and OLED over say LCOS as the display device.   I have questions as to how well the time sequential layers will work with a moving image and would there be temporal breakup-effect.

There are also a number of patents with totally different optical engines and totally different inventors (and not principles of Magic Leap) with free-form (very thick/non-flat) optics  20160011419 and 20160154245 which would fit with using an LCOS (or DLP) panel instead of the laser fiber scanning display.

I have heard from more than one source that at least some early prototypes by Magic Leap used DLPs.  This would suggest some form of time sequential focusing.

Problems I See with the “Photonic Chip” Magic Leap Showed in the June 2016 Wired picture

“Edge injection” waveguide – There needs to be an area to inject the light.  All the waveguide structures in Magic Leaps patents show use “side/edge” injection of the image.  Compare to the Microsoft’s Hololens (at right)which injects  the image
light in the face (highlighted with the green dots).   With a edge injected waveguide, the waveguide would need to be thicker for even a single layer, no less the multiple layers with multiple focus distances that Magic Leap is requires.

Lumus (at left) has series of exit prisms similar to a single layer of the Magic Leap ‘495 application Figs. 5H, 6A, 8A, and 10.  Lumus does edge injection but at roughly a 45 degree angle (see circled edge) which gives more area to inject the image and gets the light started at an angle sufficient for Total Internal Reflection (TIR).  There is nothing like this in the Magic Leap chip.

Looking at the Magic Leap chip” (right) there is not obvious place for light to be “injected”.  One would expect to see some discernible structure such as an angled edge or a some structure like in the ‘705 application Fig. 8 for injecting the light. Beyond this, what about the injecting multiple images for the various focus layers.  There is a “tab” at the top which would seem to be either for mounting or it could be a light injection area for a surface injection like Hololens, but then I would expect to see some blurring/color or other evidence of diffractive structure (like Hololens does) to cause the light to bend about 45 degrees for TIR in such a short distance.

Another concern is that you don’t see any structure other than some blurring/diffusion in the Magic Leap chip.  Notice in both the Lumus and Microsoft combiners you can see structures, a blurring/color change in the case of Hololens and the exit prisms in the case of Lumus.

Beyond this if they are using their piezo scanned laser fiber display, it generates a light spiral angular cone of light that has to be “columated” (make the light rays parallel which is shown in the patent applications) so they can make their focus effects work. There would need to be a structure for doing the columation.   If they are using a more conventional display such as DLP, LCOS, or MicroOLED they are going to need a larger light injection area.

My conclusion is that at best this Magic Leap chip shown is either part of their combiner (one layer) or just a mock-up of what they hope to make someday.   I haven’t had a chance to look at or through it and anyone that has is under NDA, but based on the evidence I have, it seems unlikely that what is shown is function.

Pupil/Eyebox

I’m curious to see how small/critical the pupil/eyebox will be for their combiner.   On the one hand they want light at a the right angles to create the focusing effects and on the other hand they will will diverse/diffused light to give a large enough pupil/eyebox which could be at a cross purpose.  I’m wondering how critical it will be to position the eye in precisely the right place.   This is a question and not a criticism per say.

What, Himax LCOS? Business Insider OCT 27, 2016 (“Magic Leap Lite”?)

I had been studying the various patents and articles for some time and then last week’s Business Insider (see: http://www.businessinsider.in/Magic-Leap-could-be-gearing-up-for-a-2017-launch/articleshow/55097808.cms) throws a big curve ball.  The article attributes KGI Securities analyst Ming-Chi Kuo as saying:

“the high cost of some of Magic Leap’s components, such as a micro projector from Himax that costs about $35 to $45 per unit.”

I have no idea as to whether this is true or not, but if true it suggests something very different.   Using a Himax LCOS device is inconsistent with about everything Magic Leap has filed patents on. Even the sequentially focusing display would at best be tough with the Himax LCOS as it has a significantly lower field sequential rate than DLP.

If true, it would suggest that Magic Leap going to put out a “Magic Leap Very Lite” product based around some of their developments. Maybe this will be more of a software, user interface, and developer device. But I don’t see how they get close to what they have talked about to date.  The highest resolution Himax production device is 1366×768.

More Observations on Stanford’s Compressive Display and Magic Leap

Both are based on greatly reducing the image content from the general/brute force case so that a feasible system might be possible.  The Stanford approach is different from what Magic Leap appears to be doing.  The Stanford System has a display panel and a “modulator” panel that selects the lights rays (via controlling the angle of light that gets through) from display panel.  In contrast Magic Leap generates multiple layers of images with different focus associated with each layer in an additive manner.   This should mean that there two approaches to things like “occlusion” where parts of an image hide something behind it will have to be different (it would seem to be more easily dealt with in the Stanford approach I would think).

A key point that Dr. Wetztein makes is that brute force light fields (ala Nvidia which hugely sacrifices resolution) are impractical (too much to display and too much to process) so you have to find ways to drastically reduce the display information.  Dr. Wetztein also comments (a passing comment in the video) the that the problems are greatly reduced if you can track the eye.  Reducing the necessary image content has to be at the hear the heart of Magic Leap as well.  In all the incarnations in the patent art and Magic Leap’s comments point to supporting simultaneously two or more focus points.   Eye tracking is another key point in Magic Leap’s patents.

One might wonder if you can eye track and if you can tell the focus point of the eyes, you could eliminate the need to the light field display altogether and generate an image that appears to be focused and blurred based on the focus point of the eye.  Dr. Wetztein points out that one of the big reasons for having light fields is to deal with the eyes focus not agreeing with where the two eyes are aimed

Conclusion

Summing it all up, I am skeptical that Magic Leap is going to live up to the hype, at least anytime soon.  $1.4B can buy a lot of marketing as well as technology development, but it looks to me that to accomplish what Magic Leap wants to do, is not going to be feasible for a long time. Assuming they can make it work (I wonder about the fiber scanning display), there is then the issue of feasibility (The Concord SST airplane was “possible” but it was not “feasible” for example).

If they do enter the market in 2017 as some have suggested, it is almost certainly going to be a small subset of what they plan to do. It could be like Apple’s Newton that arguably was too far ahead of its time to fulfill its vision or it could be the next SST/Segway.

Next time I am planning on writing about Magic Leap’s scanning fiber display.