Introduction

A Hololens 2 (HL2) was purchased directly from the Microsoft store online, and I have been evaluating its display technology for a couple of weeks. Unfortunately, this further pushed out my article on MicroLEDs that I have been working. Since the HL2 uses laser beam scanning (LBS), seen by some as a competing display technology with MicroLEDs, discussing the HL2 provides some foundation information for the coming MicroLED article.

This blog posting follows up on two articles I wrote in 2019; namely, Hololens 2 is Likely Using Laser Beam Scanning Display: Bad Combined with Worse and Hololens 2 First Impressions: Good Ergonomics, But The LBS Resolution Math Fails!. Regarding the second of the two articles, there is a slight change in the “math” because the HL2 uses dual, vertically stacked lasers per color, but the conclusion is the same because the mirror is scanning at half the speed that was reported.

As is the practice of this blog, all the test patterns are available so that anyone can repeat my results. All the test patterns used can be found on the Test Patterns page. If you want to see what is going on, you are going to want to click in the images to see them in higher resolution. Even these images are typically scaled down by more than 50% of what the camera took for web viewing.

The Laser Beam Scanning Display – No Color “Rainbow” Pictures . . . Yet

I have taken thousands of photographs of the HL2 displays. Most of these pictures were experiments to get the images that best conveyed what the eye sees as well as to capture things the eye cannot see decipher the scanning process from the images.

Yes, there are problems with uniformity and color “rainbows” across the display related to the waveguide and other optics, it is built into the “physics” of a diffractive waveguide. The HL2 unit I have is not nearly as bad as some of the pictures I have seen posted. I suspect that part of the issue is that it is difficult to get a representative picture.

I have taken representative full-color, full-frame pictures, for use later. This article is going to concentrate on how the HL2 laser beam scanning process works. Understanding how it works helps in understanding the resolution, artifacts, and temporal (time changing) issues (including flicker) of the LBS display engine.

For the LBS tests, I am only going to use two reasonably simple test patterns that test resolution. The original patterns are on my “Test Pattern Page” and are shown below (click on the thumbnails for a full-size version):

These test patterns have “targets” in nine locations; the center, top, bottom, and four corners that are based on the well-known USAF-1951 Standard Target. The name of the target locations is included in the pattern.

Definition of Field vs. Frame

For scanning displays, “field” is one top-to-bottom scan of the image. A “frame” represents the whole image at a point in time. For example, old U.S. CRT televisions used “interlaced scanning” with two 60 Hz fields, each covering odd and even lines to build up one 30 Hz frame. What constitutes a “frame” is not as clear today as some display systems will switch to showing field(s) from the next frame rather than building up a complete single “frame/image.” As will be shown below, the HL2 has four (4) identifiable “fields,” each displayed at 120 Hz, but likely, the HL2 changes the image content at 60Hz or 120Hz.

Scan Lines vs. Pixels

With scanning displays, scan lines are not the same as rows of “pixels.” In the case of the HL2, the scan lines pseudo-randomly cross through what should be a single pixel. The effective vertical resolution in “pixel rows” of the HL2 is less than the number of scan lines. For this article, a “pixel” is the size of the pixel that should be displayed.

How HL2 Builds Up an Image (How the Sausage is Made)

The following subsections walks you through how the HL2 builds up an image. It is a complicated process.

120 Hz Field Rate

As widely reported, the HL2 has a 120 Hz field rate, and my tests support this conclusion. For capturing a single field/scan, the camera was set to 1/125 of a second shutter speed.

Two vertically-stacked lasers per color

The HL2 has two lasers per color that are stacked vertically, one pixel apart. Bernard Kress of Microsoft gave an excellent presentation, captured on video, about AR Optical challenges paving the road… in February 2020. In the talk at about 54:40, he mentions that the HL2 had “Two Lasers per color.” BTW, Kress is also the author of the recently published excellent AR headset optical reference book “Optical Architectures for Augmented-, Virtual-, and Mixed-Reality Headsets.”

Below is the photographic evidence of the dual vertically stacked lasers. The picture is a close-up of 2- and 1-pixel wide vertical and horizontal lines with a crop of the test pattern enlarged to match. The picture shows a single field, and the red and blue are filtered out (in Photoshop) to show the individual laser scans more clearly.

The HL2 was positioned, and the test pattern was sized such that a single-pixel wide line in the test pattern would be the same height as one scan line from one laser.

Looking at the 2-pixel wide vertical lines of the test pattern in the photograph, notice there is a thin black line separating the two vertically stacked green lasers. The overlaid small red squares are the size of a single-pixel in the test pattern.

You should also notice that the 2-pixel-wide vertical lines are zig-zagging in pairs of two pixels. If you look carefully at the red pixel size squares, you should see that each pair of 2 pixels is shifted left or right about half a pixel. Each zig or zag of 2-pixels is coming from a different direction of the scan in a single field. The lasers are being turned on in both directions of the horizontal scanning.

Bi-Directional Scanning with the Lasers Turned On

Only in the horizontal center of the display as the scans and pixels so neatly stacked. The figure below has crops near the left side and the center from a single field (one scan). Notice how the separations of the lasers are relatively even in the center of the display and then overlap, causing a bright centerline, with gaps between scans.

The bright center and gaps are caused by the lasers being turned on in both directions (“bi-directional scanning“). An odd CRT TV or monitor that only turned on when the electron beam was scanning from left to right, which results in more evenly spaced scan lines.

The figure on the left gives further evidence of bidirectional scanning. The HL2’s red lasers are slow, turning on from black/off. It can sometimes take more than a dozen pixels for them to turn on (this is true in both displays on the HL2 being evaluated). I first noticed this with cyan (blue/green), where there should be white in areas of the test patterns. In this color close-up from near the center of the display, you can see how every other scan pair is very cyan (lack of red). Where the laser pair is more cyan is indicative of the laser scanning from the black to the white (left to right in this example).

Only about 854 scan lines (not pixels) per field

With the HL2 displaying an image with pixels that closely matched the scan lines in the center of the display, it then becomes a simple matter to calculate the total number of scan lines visible by seeing how much of the test pattern is evident. Doing the math, it turns out that the HL2 has roughly 854 visible scan lines per field, counting the dual-stacked lasers as two “scans” as well as counting both directions of the scan. This is the “maximal counting” with one “cycle” back and forth of the horizontal mirror counting as four scan lines. As will be shown, the effective vertical resolution in “pixels” of a laser scanning display is less than the number of scan lines.

Curved, overlapping, variable speed, and 4-way interlaced scanning

The HL2 scanning process is, to say the least, very complex. As I wrote back in Feb 2019, “The LBS Resolution Math Fails!” the “fast” mirror is not moving fast enough by at least a factor of four to come close to Microsoft’s resolution claims for the HL2. While the HL2 has stacked two lasers per color, which would double the number of scan lines, they are running at half the horizontal scan speed as what they said, so the net result is the same. Dual-stacked lasers also cause new image problems.

Non-uniformed curved scanning

US Patent Application 2018/0255278 to Microsoft discussed interlaced scanning with dual lasers. It also talks about the beams from the two lasers crossing during the scanning process. In addition to showing the patent application’s Fig. 6, there is a colorized version on the left below. The figures are greatly simplified and only show a few lines of the scans.

The upper left part of the figure shows the scanning process for a single field/scan with the dual-stacked lasers. A scan of a single laser is a distorted “bowing” of a sine wave. The distortion of the sine wave comes from one or more mirrors, at least some of which are curved, being hit “off-axis.” A key thing to note (see yellow ovals) is that when the scanning reverses horizontally, the lower laser’s path from the prior scan will cross the upper laser’s path for the next scan.

In the upper right corner of the figure, I have included a crop of 1 field of the green scan showing the gaps between the scans on the outsides of the displayed image for a single field. The display’s scan lines are much closer together than shown in the figure, and the dual-stacked lasers noticeably overlap for more than a third of the display on each side.

In the diagram above on the lower-left corner, I have added the “interlaced field” in red and magenta. The interlaced field is shifted about 2 pixels down, in red and magenta. The interlaced field will somewhat fill in the gaps on the left and right side of the first field.

4-Way Interlacing

One slight surprise is that the HL2 is using not the common two field interlacing, but rather four variations of interlacing or field-types. The 4-way interlacing appears to be necessitated by the use of the dual-stacked lasers, which has to skip down two lines (in the center of the display) when interlacing. With the bowed bidirectional scanning process, there would still be significant visible gaps if they didn’t also have four fields.

In the figure below, I have cropped the same area on the far left side of the four field types found to date. Four white horizontal reference lines that are one scan line apart are overlaid for reference. Each of the four fields starts one laser scan width down from the next. While I have numbered the fields in numerical order from top to bottom, I would expect that they probably go in a different order to reduce temporal artifacts.

Bowed Scanning

Shown below is a full-field image (green only). On it, I have drawn three curved red lines that follow a scan line on the top middle and bottom of the image. The scan lines exhibit the bowed scanning, as shown in the Microsoft patent application. I have included in light blue, horizontal straight lines for reference.

The next figure shows crops from nine “targets” of the white on black test pattern in a single field. It gives some idea as to how the scanning varies across the display.

Four-Way Interlaced (Wobulation) – Papering Over the Cracks

In 2005 HP, then in the rear-projection TV business, used a vibrating mirror to move around a DLP image very quickly with a technique they called “wobulation.” While officially the term “wobulation” died with HP’s exit of the TV business, many in the industry still call it “wobulation.” Today there are several companies making projectors that do a 4-way shift, which has been dubbed “Faux-K” (see True 4K vs. Faux-K). The figure below shows the concept of 4-way shifting/wobulation from Kress’s book.

fthus the derisive Faux-K nickname). You simply can’t paint a 1-pixel wide line with a 2-pixel wide brush. The 4-way shifting primarily makes the image smoother and hide screen door effects.

The HL2 uses a variation of 4-way pixel-shifting/wobulation. I have captured four primary shifts below taken from four different fields. Fields 1 and 3 shift every other vertical pair of pixels left or right by 1/2 pixel with opposite zig-zag effects (look at the vertical lines under the “2” in the figure below). Similarly, fields 2 and 4 zig-zag in opposite directions. Below, in Photoshop, I have averaged together fields 1 & 2, 3 & 2, all four fields, and fields 2 & 4.

Averaging Fields 1 and 3 or fields 2 and 4 produce a reasonable image. But note, there is still a considerable zig-zag effect when averaging just two fields.

Using a flat white background shows where there are gaps or unwanted textures caused by the scanning process. The two sets of images from the center and the left side below show the four fields. As before, the various fields are averaged together to show the net effect at 30 Hz.

If you look carefully at the “All 4” images above, you will notice that it still has some wiggling. Also, in large flat shaded areas, you can still see horizontal lines/textures even when adding all four fields together. At first,

Verifying the averaging process above, I took a series of pictures at different shutter speeds (see below). When shot at 1/30th of a second, the camera is averaging four fields together (120 fields/30 = 4 fields). Notice there are still lines/textures in the image similar to the photoshop averaging. I found I had to slow the shutter speed down to about 1/8th of a second or 15 fields to get the center smooth with no wiggles and the size of the lines/textures where they would not be noticeable. The figure below shows crops from the center and center-left of a test pattern (both sides and the corners act similarly).

A close inspection of various samples of the same field type shows that the HL2 has minor variations from Field to Field of the same type. On the left are two variations. You should notice that their overall appearance is similar. But if you look carefully, you will find minor differences. For example, the two pixel wide horizontal lines point at by the red arrows have differences in intensity but are in the same location.

I have also seen some sub-pixel movement of the fields but have not identified what is going on. Perhaps there is some vibration (something to explore later).

While the pixel shifting hides some artifacts such as the “screen door effect,” it also softens/blurs sharp edges. In the case of the HL2, it is using pixel shifting to paper over the holes caused by the bidirectional scanning.

Temporal Artifacts Including 60 Hz and 30 Hz Flicker

Pixel shifting is that because it presents the whole image over time, it causes “temporal artifacts” when people move their eyes (which is constantly), the best know being flicker. With 4-way shifting (“wobulation), it takes four frames to get a mostly complete image. With the base field frequency being 120 Hz, this means there are 60 Hz and even some 30 Hz flicker components.

As I wrote in Hololens 2: How Bad Having Tried It?, the industry learned in the 1990s that scanning computer displays, with typically 200 nits should have a non-interlaced refresh of about 85 Hz. The HL2 is specified to have up to 500 nits, and the graph below suggests it should have a non-interlaced update of better than 95 Hz.

These studies resulted in ISO-9241-3 in 1992 recommendations for computer monitors. It was found that even 60Hz progressive scanning was not fast enough and that the perception of flicker also varied with screen brightness (among other factors). The ISO committee put out a recommendation based on a formula but which simplified down to about 85Hz refresh for most practical uses. See the graph below based on the ISO-9241-3 standard from the article The Human Visual System Display Interfaces Part 2 on website What-When-How.

Another issue for the HL2 is that humans are more susceptible to flicker in their peripheral vision. And it is the sides of the display that are flickering the most as the black gaps come and go with the interlaced scanning processes.

The slower temporal artifacts will give a rippling appearance. I particularly notice ripping in horizontal lines. The user will see solid areas become striped when they move their head or eyes. Many people have complained that small text is hard to read due to the rippling.

Red lasers can’t go from black/off in a single pixel or more in the center of the display

As shown earlier, there is a problem with the red lasers turning on from full black. White lines on backgrounds come out cyan, which means they lack red. The problem gets worse at the lower brightness levels on the HL2.

Below is the full-color image and just the red component from a single image. There are crops from the left and the center of the frame. Notice inside the yellow ovals that the white lines are much more cyan in the crop from the center than from the left side. In the red component image, it can be seen that there is almost no red in the center crop. The reason for this difference is due to the scanning process, which moves slowly on the outsides (the speed drops to zero when it reverses) than in the center where the beam is moving at maximum speed, and the lasers have to switch faster for the same width pixel.

Mirror Scanning Speed is 27 kHz and not 54 kHz as Previously Reported

Quoting The Verge Hololens 2 announcement article based on information they got from Microsoft during the February 2019 announcement:

The lasers in the HoloLens 2 shine into a set of mirrors that oscillate as quickly as 54,000 cycles per second so the reflected light can paint a display.

Quoting the 2018/0255278 patent application to Microsoft with my highlight:

However, current MEMS technology places an upper limit on mirror scan rates, in turn limiting display resolution. As an example, a 27 kHz horizontal scan rate combined with a 60 Hz vertical scan rate may yield a vertical resolution of 720p. Significantly higher vertical resolutions (e.g., 1440p, 2160p) may be desired, particularly for near-eye display implementations, where 720p and similar vertical resolutions may appear blurry and low-resolution

This statement suggests a problem with getting the “fast/horizontal” mirror to oscillate faster than 27kHz and thus a reason to stack the two lasers. The rest of the math in the application seems a bit “fuzzy.” It is interesting as well for saying that “720p and similar vertical resolutions may appear blurry and low-resolution” when the HL2 is lower resolution than 720P.

The HL2 has 120 fields per second. The photographic evidence shows that there are 854 (plus or minus about 4) laser scans per field counting the two stacked lasers as two scans and counting the bidirectional scan. The mirror thus sweeps 854/4 = 213.5 cycles per frame. Multiplying 213.5 by 120 Hz gives ~25.6 kHz and or about 27 kHz, including about 5% for vertical retrace.

The Field of View Checks Out As Claimed at About 43 By 29 degrees.

The main lens I used was an Olympus 25mm (prime) 4/3rds system lens with a FOV of 37.6 by 28.2 degree. The HL2 image just overfilled the camera’s horizontal FOV by about 5%.

Conclusion: Amazing Technology Used To Produce A Low-Resolution Image that Flickers

It is impressive all the technology Microsoft used in the HL2, and in particular, the precision of the laser alignment. I have been evaluating laser scanning displays (LBS) since before this blog started in 2011, including Cynic’s Guild to CES — Measuring Resolution. In 2015, I wrote a series of articles Sony’s LBS engine using Microvision mirrors in the Celluon LBS projector (see: Celluon Laser Beam Scanning Projector Technical Analysis – Part 1, Celluon Laser Beam Steering Analysis Part 2 – “Never In-Focus Technology,” Celluon LBS Analysis Part 2B – “Never In-Focus Technology” Revisit, and Celluon/Sony/Microvision Optical Path).

It has been proven beyond any doubt (see: https://www.kguttag.com/2020/05/18/teardown-shows-microvision-inside-hololens-2/) that the HL2 is using Microvision’s laser scanning mirror technology. In my 2011 and 2015 evaluations, it was evident that the red, green, and blue lasers were not perfectly aligned and that Microvision (and their 2015 partner Sony) was digitally resampling/scaling to try and align the lasers. The HL2 has orders of magnitude better laser alignment.

Still, for all the technology and likely 100’s of millions of dollars spent, the laser scanning engine produces a terrible image by today’s standards. With all the resampling and “wobulation” going on, it is hard to put an exact resolution number. In my experience, an 800-by-600 fixed-pixel display would look better and sharper than the HL2’s display.

For all the technology and money that was spent, as the saying goes, “they are still putting lipstick on a pig.”