Introduction

Apple, like all big companies, constantly files patents on concepts with no serious intention of using them in products.  Additionally, just because a patent is filed, it does not mean that the idea is practical or even that it works as intended. It is, therefore, irresponsible to say that a company will use some concept in a product based solely on there being a patent application. One should dig deeper to see if the concept first makes any sense and second if it fits with related I.P. from the company.

The patent application I am going to be discussing today, US 20190324274,  is a classic case of what I call a “non-serious patent application.” It is both impractical and would likely be unsafe to the eye. Yet, I have seen several articles indicating that Apple would use it in a product.

Apple Hiding the Ball on AR Patents?

For a company the size of Apple and with the CEO publicly saying that AR will be as big as the iPhone, Apple has very few AR-related patent applications in its name. And of the AR patent application assigned to Apple, I have yet to find but a very few what I would consider somewhat serious applications, the application that is the subject of this article not being one of them. A search of common terms related to AR will turn up less than 1/8^th the number of patent applications compared to companies like Microsoft, Oculus/Facebook, and Magic Leap.

Most of the few “serious” patents seem to come from companies acquired by Apple. Additionally, A simple search of patent applications assigned to Apple directly will not turn up all of their AR patents. For example, Apple acquired Akonia Holographics back in August 2018, but the Akonia patent applications show on the US Patent Office site as still being assigned to Akonia and not Apple. Akonia alone, only adds a few AR-related patents.

If you look only at AR patents assigned to Apple, the only reasonable conclusion is that they are either A) not seriously pursuing AR, B) taking the risk of not filing patents, or C) hiding the patent applications buy not using Apple’s name in the assignment. For example, Apple could own one or more patent-holding companies with different names. They might also be filing in foreign countries first, picking those countries where it is harder to search for patent applications.

The hiding of the AR patents could also be linked to the way Apple “hides” it profits overseas for tax purposes. A common patent-related tax maneuver is to transfer the ownership of patent offshore. Patents almost always have no value at the time of filing, and they only take on a monetary value when issued and there is a sizable market, usually many years later. Thus, the patent application can be sold to an overseas subsidiary with little or no tax consequence, then the profits from licensing would go to the subsidiary in a country with lower than U.S. taxes.

Patently Apple Confusion

Patently Apple seems to do a reasonably good job tracking what Apple is doing but misses with me on their analysis. Also, they seem to catch only/mostly patents/applications directly assigned to Apple. A case in point was their Oct. 24^th, 2019 article “Apple Invents Augmented Reality Glasses using Photochromic Lenses” based on US Patent Application 20190324274. I take particular exception to their embedded caption (with my emphasis) the statement, “Apple’s Wearable Device will use Polarized Photchomacic (sic) Lenses to Assist Augmented Reality.”

Also, the Patently Apple article and caption both say it is using a polarizer. But the mention of polarization is incidental and unlabeled in any of the diagrams and unrelated to the photochromatic darkening operation. Someone that understood what they were writing about would know that Photochromatic and polarization are an unusual combination.

Photochromatic (Darkening with light) Materials Use in AR

“Photochromatic” materials darken with applied light. The most well-known example is the Transitions® sunglasses. Most commonly, they darken in response to Ultra-Violet (UV) light, of which there are high amounts in sunlight.  

Using photochromatic lenses has been suggested by many people in the past in combination with AR. A quick search found over 700 “AR” patents that mention photochromic in addition to AR. The most common idea is to use them to darken in sunlight to make the AR image more visible. The critical problem is that the darkening effect can’t be turned off.

Hard Edge (Pixel) Occlusion – Everyone Forgets About Focus

“Hard Edge Occlusion” is the concept of being able to block the real world with sharply defined edges, preferably to the pixel level. It is one of the “Holy Grails” of optical AR. Not having hard edge occlusion is why optical AR images are translucent. Hard Edge Occlusion is likely impossible to solve optically for all practical purposes. The critical thing most “solutions” miss (including US 20190324274), is that the mask itself must be in focus for it to sharply block light. Also, to properly block the real-world, the focusing effect required is dependent on the distance of everything in the real world (i.e., it is infinitely complex).

The most common hard edge occlusion idea suggested is to put a transmissive LCD screen in the glasses to form “opacity pixels,” but this does not work. The fundamental problem is that the screen is so close to the eye that the light-blocking elements are out of focus. An individual opacity pixel will have little darken effect, with most of the light from a real-word point in space going around it and into the eye. A large group of opacity pixels will darken as a blurry blob.

Hard edge occlusion is trivial to do with pass-through AR by essentially substituting pixels. But it is likely impossible to solve for anything more than special cases of a single distance (flat) real world with optics. The difficulty of supporting even the flat-world special case is demonstrated by some researchers at the University of Arizona which is now assigned to Magic Leap (the PDF at this link can be downloaded for free) show below. Note all the optics that are required to bring the real world into focus onto “SLM2” (in the patent 9,547,174 figure) so it can mask the real world and it solves the case for everything being masked being at roughly the same distance. None of this is even hinted at in the Apple application.

Apple “will use?” — Absolutely Not

What Patently Apple does best is work as a web crawler/aggregator that organizes clips figures from Apple patents. It seems to do a poor job of understanding how the technology in the patent works and its significants. They often use language that claims Apple is going to make a product based on whatever patent application they discover. In this case, they made the bold claim, “Apple’s Wearable Device will use,” which I think is false.

I will get into the technical reasons later, but this patent makes zero technical sense. I don’t think the concept would work well. Worse yet, one of the implementations would likely violate eye safety standards for UV radiation.  In short, the concepts shown in the application are highly impractical and would never be seriously considered for a real product. It is the kind of idea people think up in a room without much thought.

The Information Nov. 11, 2019, Picks Up The Story

The Information November 11^th, 2019 Apple Eyes 2022 Release for AR Headset, 2023 for Glasses references the same patent application (US 20190324274) as part of their report on a meeting at Apple on AR. The “The Information” article is behind a paywall, but much of the content can be found in an article in The Verge, which did a report based on The Information’s article.

It was The Information’s story referencing the application, which started me looking into it. I then worked back to find the Patently Apple reference to try and figure out why The Information would link to it. Why would they reference this seeming useless application? As I have reported before

According to The Information, “Apple tells employees it could release an [pass-through] AR headset in 2022, [optical see-through] glasses in 2023.” I plan on reporting more on these claims in upcoming articles.

Playing Telephone and Marketing Spin

Sometimes I feel like I am at the end of a playing telephone (also known as Chinese Whispers) game where the message gets more garbled with each retelling. In this case, The Information states:

Additionally, Apple has explored the use of lenses for the glasses that darken when people are using AR on them, a way of letting others know the wearer of the glasses is distracted, said another person involved with the project. Apple applied for a patent for such a feature [links to US020190324274] earlier this year. Apple senior managers have told employees they believe later versions of the AR glasses could replace the need for iPhones in roughly a decade, the person said.

The Verge article repeats the “Apple has also reportedly explored lenses that darken when in operation, to signal to observers that the user is otherwise occupied” claim. Usually, AR headsets use darkening lenses/filters to make the display’s image stand out better when competing with light in the real world. The whole “darkening to signal observers,” if it is being said by Apple people, I would say is marketing spin.

UV Projector

The central concept is to use a UV light source to illuminate a spatial light modulator (such as a DLP) to projector a UV light pattern onto a polychromatic layer to form “opacity pixels.”

First, I can’t imagine any company that would be willing to put a bright enough UV projector to control a photochromatic layer in such proximity to the eye. UV will damage the eye, and there are too many ways that light from a bright projector will make it to the eye. The amount of UV light the sun puts out to make polychromatic sunglasses work is much brighter than any safety standard allows.

Second, the “opacity pixels” are going to be so close to the eye that they will be entirely out of focus.

Then there is the question of what happens if the sun is out with all its UV light? The sunlight would likely dominate over any safe amount of UV light that could be projected.  

Heating Element – Figure 7

Another issue with polychromic material is that it responds slowly when cold, something that is well-known. The application addresses with the obvious solution of adding a heating element layer as shown in Figure 7 (left).  

The heater is what I call “closing the barn door after the horse has already left” type of detail. The basic idea is worthless, so there is no saving it with additional improvements.

Transparent OLEDs Displays Won’t Work Due to Focus

I did find it “entertaining” that they include using a “transparent OLED” as a display option (see Fig. 7 above). For fun, let’s ignore that “transparent OLEDs” are not that transparent. The fatal flaw is that the eye can’t focus on an image that is less than about 10 inches from the eye.  They are not alone in ignoring the issue of focus. I can’t believe how many, seemingly technical, people think that transparent OLED would work in AR headsets.

Conclusion

It is simple-minded and wrong to say that just because a company files a patent that they are going to make a product using it. To conclude that a company is using some patent concept, one must first ascertain whether the idea is or will soon be practical. It is also helpful to see if the patent fits with other I.P. that the company has filed. Sometimes, like in the case of my investigating Magic Leap Patents in 2016, patent trail point directly to the final product a few years later.

In the case of Apple, they are either not seriously working on AR, or they are doing a good job of covering their patent trail — at least so far.  With Apple covering their patent trail, it leads to articles citing patents by Apple that are unrelated to any potential final product.