There have been a number of comments on this blog that I am very negative about Head Mounted Displays (HMDs), but I believe I am being realistic about the issues with HMDs. I’m an engineer by training and profession and highly analytical. Part of building successful products is to understand the issues that must be solved for the product to work. I have seen a lot of “demo ware” through the years that demo’s great but fails to catch on in the market.
It’s rather funny the vitriolic response from some of the people posting in the comments (some of which are so foul they have been blocked). Why do they feel so threaten by bringing up issues with HMD? There must have been over 100 HMDs go to market, some of these with big name companies behind them, over the last 40 years and none of them have succeed in making the break through to a consumer product. Clearly the problem is harder than people think and it is not from a lack of trying by smart people. Why is the “101st” attempt going to succeed?
I know there is a lot of marketing and hype going on today with HMDs, I’m trying to cut through that hype to see what is really going on and what the results will be. I also have seen a lot of group think/chasing each other where a number of companies are researching the same general technology and the panic to get to market after another company has made a big announcement. Many feel this is going on now with HMD in response to Google Glass.
Designers of HMDs have a huge number of design decision to make and invariably each of these choices result in pro’s and con’s. Invariably they have to make trade-offs that tend to make the HMD good for some applications and worse for others. For example, making a display see-through may be a requirement for augmented reality, but it makes them more expensive and worse for watching moves or pictures. For immersive Virtual Reality the design may want a wide field of view (FOV) optics which for a given display resolution and cost means that you will have low angular resolution making it bad for information displays.
To begin with I would like to outline just some of the basic display modality options:
See-through – Usually for augmented reality. It has the drawback that it is poor for watching movies, pictures, and seeing detail content because whatever is visible in the real world becomes the “black.” The optics tend to cost more and end up trading image quality for the ability to see through. Also while they may be see-through, they invariably have to affect the view of the real world.
Monocular (one-eye) – A bit harder for people to get used to but generally less expensive and easier to adjust. People usually have one “dominant eye” and/or good eye where the one display should be located. A non-see-through monocular can provide a bit of a see-through effect, but generally the display dominates. Monocular HMDs support much more flexible mounting/support options as they don’t have to be precisely located in front center of the eyes.
Binoculars (both eyes) – Generally supports better image quality than monocular. Can more than double the cost and power consumption of the display system (two of most everything plus getting them to work together). Can support 3-D stereoscopic vision. The two displays have to be centered properly for both eyes or will cause problems seeing the image and/or eye strain. More likely to cause disorientation and other ill effects.
Centered Vertically – While perhaps the obvious location, it means that the display will tend to dominate (or in the case of non-see through totally block) the user’s vision of the real world. Every near eye display technology to at least some extent negatively affect the view of the real world; even see-though displays will tend to darken and/or color change, and/or distort the view.
Above and Below – Usually monocular displays are located above or below the eye so that they don’t impair forward vision when the user looks straight forward. This is not optimal for extensive use and can cause eye strain. Generally the above and below position are better for “data snacking” rather than long term use.
Within the above there are many variations and options. For example, with a see through display you could add sunglasses to darken or totally block the outside light either mechanically or with electronic shutters (which have their own issues), but they will still not be as optimal as a purpose built non-see through display.
Then we have a huge number of issues and choices beyond the display modality that all tend to interact with each other:
Cost – Always an issue and trade-off
Size and Weight – A big issue forHMDs as theyare worn on the head. There are also issues with how the weightis distributed from front to back and side to side
Weight on the person’s nose – I call this out because it is a particularly problem, any significant weight on the nose will build up and feel worse over time (anyone that has had glasses with glass rather than plastic lenses can tell you). Therefore there is generally a lot of effort to minimize the weight on the person nose by distributing the force elsewhere, but this generally makes the device more bulky and has issues with messing up the user’s hair. The nose bridge when use is generally used to center and stabilize the HMD. Complicating this even more is the wide variety of shapes of the human head and specifically the nose. And don’t kid yourself thinking that light guides will solve everything, they tend to be heavy as well.
Resolution – Obviously more is better, but it comes at a cost both for the display and optics. Higher resolution also tends to make everything bigger and take more power.
Field of View (FOV) – A wider FOV is more immersive and supports more information, but to support a wide FOV with good angular resolution throughout and support high acuity would require an extremely high resolution display with extremely good optics which would be extremely expensive even it possible. So generally a display either as a wide FOV with low angular resolution or a narrower FOV with higher angular resolution. Immersive game like application generally chose wider FOV while more informational based displays go with a narrower FOV.
Exit Pupil Size – Basically this means how big the sweet spot is for viewing the image in the optics. If you have every used an HMD or binoculars you will notice how you have to get them centered right or you will only see part of the image with dark ring around the outside. As the FOV and Eye relief increase it becomes more and more difficult and expensive to support a reasonable exit pupil.
Vision Blocking (particularly peripheral vision) – This can be a serious safety consideration for something you think would wear by walking and/or driving. All these devices to a greater or less or extend block vision even if the display itself it off. Light guide type displays are not a panacea in this respect either. While light guide displays block less in front of the user, they have the image coming in from the sides and end up blocking a significant amount of a person’s side peripheral vision which is use to visually sense things coming toward the person.
Distortion and Light Blocking – Any see-through device by necessity will affect the light coming from the real world. There has to be a optical surface to “kick” the light toward the eye and then light from the real world has to go through that same surface and is affected.
Eye relieve and use with Glasses – This is an issue of how far the last optical element is away from the eye. This is made very complicated by the fact that some people wear glasses and that faces have very different shapes. This is mostly an issue for monocular displays where they often use a “boom” to hold the display optics. As you want more eye relief, the optics have to get bigger for the same FOV, which means more weight which in turn makes support more of a problem. This was an issue with Google Glass as they “cheated” by having very little eye relief (to the point that they said they were not meant to be used with glasses
Vision correction – Always and issue as many people don’t have perfect vision and generally the HMD optics want to be in the same place as a person’s glasses. Moving the HDM optics further away to support glasses makes them bigger and more expensive. Building corrective lenses in to the HMD itself will have a huge impact on cost (you have to have another set of prescription lenses that a specially fit into the optics). Some designs have included diopter/focus adjustment some many people also have astigmatism.
Adjustment/Fit – This can be a big can of worms as the more adjustable the device is the better it can be made to fit, but then the more complex it gets to fit is properly. With binocular displays you then have to adjust/fit both eye which may need moving optics.
Battery life (and weight) – Obvious issue and they are made worse dual displays. At some point the battery has to be move either to the back of the head (hope you don’t have a lot of hair back there) or via a cable to someplace other than the head.
Connection/cabling – Everyone wants wireless, but then this means severe compromises in terms of power, weight, support on the head, processing power (heat, battery power, and size).
How it is mounted (head bands, over the head straps, face goggles) – As soon as you start putting much stuff on the head a simple over the ears with a noise bridge is not going to feel comfortable and you start to have to look to other ways to support the weight and hold it steady. You end up with a lot of bad alternatives that will at a minimum mess with people’s hair.
Appearance – The more youtry and do on the head, bigger and bulkier and uglier it is going to get.
Look of the eyes – I break this out separately because human’s are particularly sensitive to how people’s eye’s look. Many of the HMD displays make the eyes look particularly strange with optical elements right in front of the eyes (see below).
Storage/fragility – A big issue if this is going to be a product you wear when you go out. Unlike you cell phone that you can slip in your pocket, HMD don’t generally fold up into a very small form factor/footprint and they are generally too fragile to put in your pock even if they (and with all their straps and cables they may have) would fit.
Input – A very big topic I will save for another day.
If you take the basic display types with all the permutations and combinations of all the other issues above (and the list is certainly not exhaustive) you get a mind boggling number of different configurations and over the last 40 years almost every one of these has been tried to a greater or lesser extent. And guess what, none of these have succeeded in the consumer market. Almost every time you try and improve one of the characteristics above, you hurt another. Some devices have found industrial and military used (it helps if there is not a lot of hair on the user’s head, they are strong enough to carry some weight on their head, they don’t care what they look like and they are ordered to do the training).
In future posts, I plan on going into more detail on some of the options above.
One last thing on the comment section; I’m happy to let go through comments that disagree with me as it helps both me and others understand the subject. But I am not going to put up with foul language and personal attacks and will quickly hit the “trash” button so don’t waste your time. I put this under; if you can’t argue the facts then you trash the person category and I will file it accordingly.
Karl, thank you for the second chapter of the HMD saga. I too was surprised by the hostile comments from some readers. Such comments can only be considered naïve and subjective. As a working engineer myself with about 15 patents, I’m grateful to hear your perceptions, well knowing you have 140+ patents and a few decades of engineering experience. I doubt many of your naysayers can say the same.
To touch on one matter in the Augmented Reality “See Through” section, I read that Microsoft purchased about 80 patents from the Osterhout Design Group. If you look at some of the Osterhout patents and videos (on YouTube), you’ll notice a technique to shade or block ambient light (perhaps using LCOS) on a pixel by pixel basis. I feel this is an elegant solution to the “See through” problem that you indicate. The ambient light blocking HMD in operation could then shade or block regions of ambient light and illuminate those same regions with HMD synthetic light. This would reduce background light and increase image contrast. Of course it’s not perfect. Such a technology would likely further increase cost, weight, size, and complexity.
I believe Osterhout are using liquid crystal displays, essentially an SLM system to achieve their dark pixels.
At the moment we’re using a new electrochromic polymer to do the same thing but the ideal solution would be to use a photochromic layer that turns black and has a narrow spectral sensitivity.
I assume you are saying that you are in effect using a secondary display to try and selectively blacken the background/real world. This is at best a tricky problem because of parallax and focus because the image and the layer that does the darkening are not in the same physical location and the image being generated usually has its focus moved out optically.
Could you explain a bit more?
At the output of the waveguide behind the holographic layer and a thin plastic layer we pattern the electrochromic polymer and the appropriate elctrodes, we’re going for a lower resolution here than the actual display just to make things easier to start off. It’s essentially acting as a low res greyscale display, the bigger “pixels” here means the issue is a bit different to the depth difference when viewed at an angle problem because either the colour has a a shadow border around it or it has ambient blocking inside its border.
Interesting and conceptually simpler to local dimming with LEDs (for example/definition http://www.rtings.com/info/what-is-local-dimming). It would seem you would want the dimming diffuse/gradual to reduce issues with alignment and to prevent the eye seeing the edges induced by the dimming particularly if it is low resolution. The fact that the generated image is generally focused far and the light blocking layer will be very close to the eye works would seem to work in your favor.
I would think a lot of people have thought of doing this before, the trick is in making it work well. There is what I call “Heisenberg’s uncertainty principle” problem in that whatever you do to affect the light you want to block also affect the light you want to get through. This is of course a big problem with LC materials most of which that “work” by controlling the polarization of the light and thus much polarize any light they want to control. So you start off with blocking over 50 percent of the real world light plus whatever structures are there (control lines and transistors) including their diffraction effects as well as the optical effects of the various layers making up the LC including the glass. I would think you would have some similar issues with electrochromic polymers and the various layers.
Sorry the form won’t let me reply below your last comment.
The physical set-up is easier than an LED backlight LCD dsiplay and more like a proper OLED panel.
The electric field extends slightly laterally beyond each patterned surface ITO electrode so this darkens the gap between pixels to avoid the screen door effect in reverse.
The polymer doesn’t affect polarisation and the layers can be reasonably thin. We’re getting about 75%-3% transmission maxima which is nice. It should work for the 2-in-1 augmented reality and virtual reality from a single device quite nicely.
The big issue with this is that the graphics pipeline must now be considered to have RGB+shadow+transparency or clear alpha channel.
It will be more complicated for light fields as the position of the eyes as well as their gaze vectors must be calculated in real time to properly shadow a holographic image formed beyond the glasses with a flat plane of greyscale pixels.
If you can say, are you using passive matrix, active matrix, or direct drive? It kind of sounds like you are directly driving patterned ITO which would severely limit the resolution. The next questions would be the switching speed of the material and the drive voltage. A lot of the electrochromic materials seem to need a high voltage which has issues with controlling inexpensively.
The ~75% throughput seems very good but of course this is multiplicative with whatever the light guides block. It also makes me wonder about the voltage (per above), because at least with the little experience I have had with non-polarizing electrochromics, they require high voltages (50V to 100V or more).
It is interesting that you are thinking about the parallax effects for light fields. While light fields are technically very interesting they would seem to be very far out in time for any high volume application (they would seem to require extremely high pixel resolutions of the display device to support).
For “2 in 1” AR/VR it would first be used as simple shutter (AR or VR mode) and then speed and resolution are not issues, but voltage will still be a bit of an issue.
Your electrochromic polymer sounds like an interesting area to explore. I too am curious about the switching rates you are getting with your electrochromic polymer. Recognizing frame rates of 30 to 120 frames per second for displays, does the electrochromic shading mechanism keep up?
Also, does electrochromic materials and photochromic materials degrade with usage? I know that “photochromic dyes” degrade with usage. The photochromic molecules lose their ability to change their shape and alter light. I’m sure this is not a high priority issue as of now in HMD development, but perhaps some readers can share their experiences with color changing or optically active materials regarding endurance.
The polymer takes about 2V to switch fully and is reasonably linear for following voltage at being greyscale. This was direct drive to prove the principle.
Light fields are very close to being mainstream, Magic Leap will have their demo devices this summer and Ostendo (shipping Q1 2016) have already shown theirs. SeeReal has their IP and knowhow ready to licence. Samsung have filed for similar tech to the SeeReal method.
We’re working on two other methods, one of which uses a standard LCOS and some tricks at the waveguide outcoupling, the other needing a half-silvered mirror and using a DMD like TI DLP. I hope to have the DMD one as a demo at the Augmented World Expo, the LCOS type will take longer.
The trick to getting the good light field is to both interpolate and multiplex it, having the highest resolutions appearing in near focus, nearest the imperative horopter and within the foveal region.
Your polymer material has some very good characteristics. Like Ken, I am curious about the switching speed. Since it is a polymer, I’m assuming it might be susceptible to damages by UV light (which would be say OK for a headset but not for say use in a car or window). I’m also interested in the cost (for HMD you can afford very high cost per square mm materials).
I think you are being at least a bit optimistic about light fields and/or your definition of mainstream is different. At a minimum it needs a lot more resolution and more complex optics than conventional displays for the same effective resolution. So for HMD to me this is flying before they learn to walk as HMDs with conventional displays have not make the mainstream yet.
Sounds interesting about trying to concentrate the resolution with a light field, but it also sounds complex and I wonder about how it works with the eyes moving and normal movements of the HDM on the head.
The workable light fields today have had to greatly sacrifice resolution. All the microdisplay technologies are pretty close to their physical limits on resolution. DLP can only make the mirrors so small, LCOS is limited by the lateral fields between pixels, and OLEDs have limits on the transistor and circuitry size although it has possibly the most room for improvement.
I’m a bit confused by what you are saying in your “other methods” paragraph. Are you just talking about the basic display and then applying your dimming on top of it or is the DMD or LCOS integral to your method?
I am of the opinion that a laser source display largely gets rid of the problems listed of exit pupil size and eye relief.
I find that well made holographic waveguides don’t distort the ambient scene other than a very slight dullness and only look strange to another person at a precise angle to reflect the 3 recording wavelengths from an ambient white light source.
I believe that the mounting of the light engine system above the eye, either with a microOLED display or a full LCOS laser engine in a linear configuration along the top of the glasses should leave the glasses frames much narrower at the temples.
Also the first limit to a holographic waveguide’s FoV seems to be in the direction of its TIR transmission so this vertical transmission of the image gives greater horizontal FoV.
At the moment for extra battery we use hot-swappable ones behind the ear and around the back of the head, it works pretty well.
After some research in wireless magnetic resonance power transmission, I conclude that the use of better geometries and active magnetic field shaping means the ability to send power over a distance several diameters larger than that of the transmitting coil and this will be useful for other power sources for instance worn embedded in a necklace or eventually in the pocket.
To remove the processing power and associated weight and power from the head, it seems that WiGig will be suitable for sending the data to and from the glasses, with the more powerful Snapdragon module as essentially a phone and a weaker module to provide the throughput to the displays on the head unit.
Besides this well written article, there are two important areas that have been left out: the social etiquette of constantly active cameras and of their ability to track to the world at large.
We need to enter an era of accepted veillance for these glasses to be accepted, people may even be suspicious of the ToF and structured light types.
I saw a radical proposal a while ago to modify the WiFi ad (WiGig) set-up to allow for depth-mapping in the radio spectrum. This may be more acceptable and is certainly more subtle.
We all realise the issue of mapping in this industry and no-one can produce reliable compact solutions with an effective range larger than a few metres and even then pre-scanning is desirable. Therefore the way forward is the use of massive photogrammetric databases like Google Streetview and a single RGB or basic system on the local user device.
Jack- Halo AR
I have a comment about desktop VR [sometimes called “Fishtank VR], which may also relate to HMDs. I have worked in the field of Computer Aided Design for about 25 years. At the beginning of my career, I recall seeing a LCD shutter glasses demo using an Atari computer. I expected to see this technology explode into the world of CAD. But, it did not.
The technology to give a user a display on their desktop, using their existing display, seemed mature a decade ago. And even if rendering lag time was not fast enough for a HMD, for an object apparently floating in a fish tank, a lag is acceptable.
Around that time, I recall saying something like, “Fish Tank VR costs around $3,000 in hardware, and $100,000 in software development to get the system to talk to your CAD program.” In retrospect, I was estimating very high in the hardware costs, and low in the software costs. Clearly if the software vendors won’t support it, it won’t happen.
So, why did stereoscopic display [with or without head tracking] never take off in the world of CAD? Here are some ideas, which may also apply to HMDs.
Perhaps the real-time rendering of models as you rotated them was good enough with a 2D display, so that there wasn’t a significant benefit to using stereoscopic displays. I went to see one of the first public games to use a HMD, and was disappointed in the experience. But, it was not that the technology wasn’t good enough. Rather, I realized the reason is that I had spent years looking at a flat 13″ monitor, without real-time shading or view rotation. I had developed the skill to form a 3D model in my head. Now, if my supervisor asked me to view something on his CAD display, I often had to ask him to zoom out, and do a hidden line removal, for me to get oriented. So, the need for an improved display was certainly there. But, for most of my work, stereoscopic would not have been a benefit.
At one point AutoDesk added a stereoscopic option to their free drawing viewer. But, in a later version they omitted it. Apparently the demand was not there to keep it in.
There are other technologies that had been eagerly awaited, only to be disappointing in their real benefits once the technology is viable. One example is voice recognition, which is useful for phone calls while driving, but is mostly a way to annoy us by not having a human receptionist answer the phone. Another example is video phone calls. Nearly everyone has the equipment to do, but it is regularly done by only a small percentage of people.
For the HMD, I think one area should have exploded as soon as the Google Cardboard (or one of its predecessors from years earlier) became available. Architects are skilled at looking a a blueprint, and forming a view of the building in their mind. Their customers don’t have that skill. So, a VR display is an ideal way for the customer to view the building. The cardboard style of implementation is so cheap that architects can even send them to a customer for free. But, as of yet, there is no way to view a 3D CAD model on Google Cardboard.
In high school, around 1980, I experimented with creating stereoscopic slide shows, and so I am definitely a person interested in the technology. But, I have not been able to economically justify spending the extra $200 for a stereoscopic display for my CAD station.
The above hints that HMD’s are another technology that will not be wide spread. The fact that stereoscopic TVs are not common, is another hint that HMD’s will not be widely used.
An immersive experience is certainly a nice thing to experience. If the technology for a very high quality HMD comes down to $100, then I suspect it will find a market. But, I doubt it can with its current state of hardware and cost.
Thanks Joe, lots of good information. I have seen a lot of similar things in my time working in display and display devices since the late 1970’s.
Things get a lot trickier when you put them right near the eye. There are a long list of failures in HMD, many that failed so badly that few know of them. Sony must have made at least 5 major attempts alone.
Your analogy to 3-D glasses in the home I think is particularly relevant. Few people don’t watch TV in the home sitting upright in a chair centered on the TV and the glasses are just in the way.
As I have been writing, while the displays can be a challenge to meet the cost and quality, the devil is really in all the human factors issues. HMD is a onion with many layers.
What about using your smartphone with one of the headset frames like Durovis or just making your own and then using something like the Splashtop app to mirror your desktop to the phone.