Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
Apple Vision Pro’s (AVP) Image Quality Issues – First Impressions
Speaking at SID LA One Day Conference Feb 23, 2024
As it is one week away, I want to mention again that I will speak at the SID LA One Day Conference on Feb 23, 2024. The main topic of the talk will be “Interesting Mixed Realty Things I Saw at CES and the AR/VR/MR conferences,” but I will likely include some of my Apple Vision Pro experience.
Introduction
I often say and write, “The simple test patterns are often the toughest for display systems to get right because the eye will know when something is wrong. If a flat white image is displayed and you see color(s), you know something is wrong. Humans are terrible judges of absolute color, including various white color temperatures, but the eye is sensitive to variations in color. As will be shown, the Apple Vision Pro (AVP) fails the simple (mostly) white display test. While not as horrible as some other headsets, you would never buy a modern TV or computer monitor with such poor white uniformity.
For testing resolution, the simplest thing to do is put up “line pairs” and see if you can see the right number of lines and whether they are blurry. Once again, as will be shown, the AVP has problems. The main test pattern for today will combine a mostly white image with a series of lines on it to test for white uniformity, a series of vertical and horizontal lines. The AVP has some serious problems displaying even modestly high-resolution content that was expected based on basic sampling theory (discussed in Apple Vision Pro (Part 5A) – Why Monitor Replacement is Ridiculous and Apple Vision Pro (Part 5C) – More on Monitor Replacement is Ridiculous), but has some “unusual” worst than expected behavior due to processing by the AVP.
Simple 2-D images, as often occur in “simple office applications,” are often the most challenging for the AVP, or any headset, to present as stationary flat objects in 3-D space. In addition to dealing with 3-D translations, the AVP’s optics are highly distorting, and the distortion is a function of where the eye is located and pointing. The result is that every pixel in the 2-D image must be resampled at least once, if not more than once, resulting in an inevitable loss of fidelity even of 2-D images much lower in resolution than the AVP’s display.
Anyone with even a basic knowledge of digital image and signal processing should know they are fundamental problems; it’s not that the AVP is doing something particularly wrong (although there is also some “wrong” behavior), but rather, it is an impossible problem to solve well, given the need for optical correction, basic sampling theory, and the AVP display resolution is lower than the eye’s resolution.
Distorting Optics
iFixit was kind enough to lend me the AVP, the display, and pancake optics from iFixit’s Vision Pro Teardown Part 2: What’s the Display Resolution? and a Meta Quest 3 display and pancake optics (which are similar to the Meta Quest Pro optics) from Meta Quest 3 Teardown and the Future of VR Repairability. I extracted the lenses from their housing to get a picture through the optics of a MacBook 14 screen of a spreadsheet grid (right, taken from “far away” from the eye position). Since the circular polarizers were glued to the display devices. I removed a circular polarizer from a pair of REALD-type 3-D glasses (see here for how REALD glasses work). The picture (right) shows the setup taken from “far” away (eye-view a bit later).
For more on Pancake Optics and why I needed to add the circular polarizer (a linear polarizer plus a quarter waveplate), see Apple Vision Pro (Part 4) – Hypervision Pancake Optics Analysis and Meta (aka Facebook) Cambria Electrically Controllable LC Lens for VAC? (which includes diagrams explaining how pancake optics work). As an interesting aside for the optics nerds, the AVP and MQ3 optics require oppositely left and right-hand circularly polarized light (fortunately, REALD-type glasses come with one of each).
iFixit also took a picture of the AVP’s OLED display with the optic removed (left), which shows how much the image must be pre-corrected due to the distortion of the optics.
Below are pictures through the AVP (below left) and MQP pancake optics on top of a 14″ MacBook Pro M3 Pro with a spreadsheet with a square grid. The camera is close to the optics (widest FOV). The MacBook pixels are about 13.33 times bigger linearly than the AVP pixels, or ~178 AVP pixels will fit inside a single MacBook pixel. Also, the distances from the display to the optics were not exact. So, the images below only give a rough idea of the optic distortion. If you click on the image, you will be able to see that red, green, and blue colors are separating (chroma aberrations). Since the Macbook pixels are huge compared to the AVP, multiple AVP pixel chroma aberrations exist in roughly the outer 1/3rd of the FOV.
The AVP Micro-OLED display is about 1.1 inches wide, whereas the MQ3 Display is ~1.8 inches wide, so the AVP must be magnified by about 1.6x more for about the same FOV. Thus, the letter “H” looks bigger via the AVP’s optics. We are looking for the distortion in the lines and the rate of change in the size of the letters.
The image on the right overlays the AVP’s distorting lines (in red) on top of the MQ3. Even though the AVP magnifies by about 1.6x more, the distortion seems similar, which is a remarkable accomplishment, but both are still highly geometrically distorting, like almost all VR optics.
To support the wide FOV with a relatively (to the Meta Quest Pro and Quest 3) small display, Apple has developed a more radical approach of curving the quarter waveplate and having a concave lens on the eye side of the optics (below left, from an interesting analysis by Hypervision). In contrast, the Meta pancakes (below right) have a flat quarter waveplate and convex lens on both the eye and display sides.
The Apple design with the concave surface is thought to require eye-tracking correction to work without significant distortion, including pupil swimming and color problems. If the eye tracking becomes “confused” by, say, a person wearing glasses or by closing their eyes to a slit with just enough to see, the displayed image can become “unstable” in geometry and color.
By definition, quarter waveplates (QWP) are color=wavelength dependent, and they are also significantly affected in color and affect polarization by the angle of the incident light dependent. Having a curve, QWP is necessitated by the optics design.
There is no free lunch with digital pre-correction; the resampling to remove the distortion comes at the expense of resolution. Display pixels in the center of the FOV are less magnified, while pixels are more magnified, moving out from the center. A thin line might be the size of one pixel in the center of the FOV and will be less than 1/3rd the size of a pixel on the outer part of the FOV.
I discussed the issues of optical distortion in Apple Vision Pro (Part 5B) using the Meta Quest Pro as an example. In the best case, the distortion correction can be done in combination with rendering and 3-D mapping, so the resampling resolution loss is only taken once. But often, resampling is done more than once with bitmap images when represented in 3-D space for practicality and simplicity of software. I don’t know if this is the case for the AVP, but I suspect it. The first resampling is to a larger image size, followed by the resampling into 3-D space. In the case of the AVP, there are clearly different resamplings for the foveated and non-foveated regions.
Eye Tracking Correction
Apple is using eye tracking to correct the optics in addition to foveated rendering, and most of the time, thanks to eye tracking and processing technology, the user will be unaware of all the dynamic corrections being applied. Occasionally, the eye-tracking-based rendering can go very wrong, as I showed last time in Spreadsheet “Breaks,” The Apple Vision Pro’s (AVP) Eye-Tracking/Foveation & the First Through-the-optics Pictures. The AVP can display bizarre results when the enhancements are combined with foveated rendering.
While the native spreadsheet caused dramatic problems I wrote about previously, I have seen similar eye-tracking artifacts with some static bitmaps (to be shown in a future article). Eye tracking and foveated rendering are clearly being done with bitmap images.
One problem I have seen with the AVP in both cases is that the Foveated portion has generally had “contrast enhancement,” with the unwanted side effect of not preserving average brightness, thus making the foveated region boundary visible. There is also increased aliasing scintillation (wiggling) in the foveated rendered area, as would be expected as it is rendered in higher resolution/sharper.
To be fair, most of the time, the foveated rendering does a good job. But it can fail either constantly with some images or just occasionally with others. Whether the failure is visible can depend on the image’s source (native or, say, a mirror of the MacBook Pro).
Eye movement involves translation and rotation; thus, the eyes will look through the optics at different places and different angles. This change in location and angle will cause the optics to behave differently, which would typically cause “pupil swim,” which is a varying distortion (wobbling) with eye movement. The angled look through optics will also cause chromatic aberrations (color fringing). The AVP’s pancake optics will also cause large area color variations with eye movement. Overall, the AVP seems to do a good job of digitally removing pupil swim and chromatic aberrations.
Almalence demonstration of pupil swim and their robot-controlled camera for developing eye tracking-based optics correction
The company Almalence, which I met with at both CES (in the PixMax booth) and the AR/VR/MR (in their own booth), develops software for correcting various optical problems caused by eye movement based on eye tracking. They have even demonstrated the ability to improve resolution, which I saw using PixMax (and I have heard independent reports saying it works well in practice and not just demos). Almalence has developed an eye-tracking correction for several different headsets. The video (left) demonstrates the “pupil swim” issue with before and after views. Almalence uses this “eye simulator” to develop its eye-tracking-based optical correction.
AVP Eye Tracking is a “Must Have for the Optics to Work – A “Nice to Have” for Selection
While all the marketing attention is on using eye tracking for input selection, eye tracking is critical to generating a good image, more so than prior optics. This also helps explain why very specific and characterized inserts are required for vision correction, even though there is enough room and eye relief for glasses with small frames to fit.
In my experience and use, the eye and hand tracking-based selection as they currently work are “nice to have” as a secondary selection method. But you really need a trackpad or mouse and keyboard to do serious work. Yes, it is good to have the ability to select without needing another physical device. Still, it can also be a terrible time-consuming nuisance as the main/only input device. With a physical device, your eyes will naturally look ahead as you click, but with the AVP, this will cause you to click on the wrong thing. Recovering from an inadvertent eye or finger movement can be a pain to undo and then do what was desired. Additionally, it it simply not accurate enough to pick small items.
AVP’s FOV is Highly Variable Based on Eye Distance but with only a Very Small Change in Magnification
Below are two pictures taken through the AVP’s optics for the left eye showing the FOV using the Zeiss optical inserts using the 25W face adapter (see Reddit topic Apple Vision Pro Light Seals decoded) and as close as possible to the optics with no face adapter. For the optics inserts to correctly correct vision correction, they must maintain a Vertex Distance (distance from the eye to the lens), resulting in a deeper face adapter usually being recommended if you order inserts. As has been widely reported, the AVP’s FOV increases dramatically if you remove the light seal and move your eye as close as possible to the optics (below right).
Both pictures above were taken with the light shield and optical inserts removed, as the light shield would mechanically interfere, and the insert would mess up the camera optically. The camera was moved on a tripod with a “macro focusing rail” to position the camera to approximate the FOV as seen by my eye (and why there is the spreadsheet grid with a “ruler” on it.
Interestingly, while the FOV changes dramatically, the magnification between the two images increases by only about 1% (1.01 times) as the camera/eye moves closer (see inset on the above right picture).
Through the Optics Image
The picture below shows roughly the FOV I see with the Zeiss inserts. The picture was taken by a Canon R5 with a 16mm lens using the camera’s 9-way pixel shift to produce a 400 mp initial image. That picture was then scaled down by a factor of 3 linearly (click on the image below to see the ~45mp image). The test pattern (in lossless PNG format) can be found on my Test Pattern Page or by clicking on the image on the right.
The test pattern has 1920 by 1080 pixels or just over half the resolution of each AVP OLED display (according to iFixit, the lit area totals 3660 px by 3200 pixels). Since the spreadsheet doesn’t fill the FOV, the 1920 horizontal pixels in the test pattern are mapped into very roughly 3000 AVP pixels of varying sizes due to optical distortion correction. The AVP does a very good job of correcting the geometric distortion of the optics overall.
Color Uniformity Issues
As I wrote in the Introduction, simple, mostly white images test a display color uniformity (known as “Color Purity” back in the days of CRTs). The camera is more “objective” because the human visual system dynamically readjusts colors both from image to image and within a single image, so the camera will make the problem look worse than it may appear to the eye. Still, there is definitely color uniformity problems with the AVP that I see with my eyes as well. There is a cyan ring (lack of red) on the outside of every image, and the center of the screen has splotches of color (most often red/pink).
The amount of color variation is not noticeable in typical colorful scenes like movies and photographs. Still, it is noticeable when displaying mostly white screens, as commonly occurs with web browsing, word processing, or spreadsheets.
The size and shape of the outer cyan ring and center red splotches will vary with how close the eye gets to the optics (see earlier picture comparing FOV sizes based on eye distance). It is also known that the AVP’s eye tracking is used to try to correct for color variation. I have seen some bizarre color effects when eye tracking is lost.
Close Up Full Resolution Crop Showing Center Details
The image below is a crop from the center of the original 400-megapixel picture. I tried to pack a lot in this image, including some pieces of the source image scaled up to about the same size as the image, a view through a 50 mm lens (with 3.125 times the center resolution of the original 16mm lens), which was used to estimate the FOV of each center pixel, plus some highly magnified overlays show details in the lines in the test sub-patterns.
A very useful feature of the AVP is what I call the “eye-tracking cursor,” what Apple calls the eye pointer, which is available in the AVP’s “Accessibility menu.” I also modified the pointer to have a red ring to help it stand out. The cursor can be turned on and off with a triple click on the crown dial. This cursor is particularly important when taking pictures to know where the AVP thinks the “eye” (camera lens) is pointing. It can also be useful when using eye tracking as a selection device. For this first set of pictures, the eye tracking with the cursor in the center of the screen where I wanted it and confirming that the eye tracking was not “lost.”
About 44.4 pixels per degree (PPD) in the center of the image – Gives ~20/30 vision in the center and worse elsewhere
I have been comparing and scaling high-resolution images with a 50mm lens with a narrow FOV where pixel boundaries are clearly visible and fitting them to the images of the much wider FOV 16mm lens for the purpose of determining the pixels per degree in the center of the AVP screen. The result I get is that there are about 44.4 pixels per degree (PPD) in the center of the image.
Having ~44.4 ppd gives (confirmed by looking at a virtual Snellen eye chart) about 20/30 vision in the center. This is the best case in the center of the screen, directly, not through the cameras, which are worse (more like 20/35 to 20/40). The resolution drops if you look beyond the center 1/3rd of the FOV, even with eye-tracking foveated rendering. With the AVP, you have somewhat poor vision, which it seems to try to compensate for by defaulting to making everything bigger (more on this in a bit).
The problems of resampling
From the 50mm lens shot, I made “pixel rulers” (rows and columns of red lines) to show the pixel boundaries versus various features in the test pattern. A magnified close-up of the higher-resolution image and the rulers is shown in the lower right corner labeled 1c.
Across the whole test pattern are sets of four lines, followed by a gap of two pixels and then four more lines. If you look at inset 1a, you will notice that the AVP has turned both sets of four lines into only three lines each. If you look at the longer set of these lines, for example, under the large #1, you will see the lines “wobbling” in the gaps and spacing but always, at best, three lines. These lines are constantly wiggling even if you hold your head steady. If you look at the four vertical lines to the right of the large #1, they are barely distinguishable as multiple lines.
The same 4-lines becoming 3-lines happens with the center test target. See the magnified section 1b above. As should be expected based on sampling theory, it takes more than two times the resolution of the display to represent arbitrarily oriented lines in 3-D space. Not shown in the still pictures is that everything “scintillates” (display pixel size flashes) and wiggles with any microscopic or macroscopic head movement. Even when one moves closer to so, there are well more than two pixels in the AVP’s display for every pixel (and above the two times the base “frequency” of the lines) in the test pattern such that there is clearly more the right number of lines being displayed, but there is still scintillation and wiggling.
Computer-generated images with sharp edges, including everyday applications like word processing, simple presentation graphics and charts, and spreadsheets, are very hard to reproduce when locked into 3-D space (see the Appendix for more information).
Foveated Rendering
Returning to the original full camera image above, a large dash line square roughly indicates the foveated rendering boundary. The image below takes a full-resolution crop showing a horizontal boundary (2a) and a vertical boundary (2b).
Looking at the two sets of 4 lines, the display’s resolution due to optical distortion and resampling has already dropped to where there are closer to 2 lines distinctly visible from the original 4. So, even without foveation, the resolution is dropping by this point in the display.
The AVP’s Make Everything Big and Bold “Trick”
The AVP processes and often over-processes images. The AVP defaults make everything BIG, whether they are AVP native or Macbook mirrored. I see behavior as a “trick” to make the AVP’s resolution seem better. In the case of native windows, I had to fix the window and then move back from it to work around these limitations. There are fewer restrictions on MacBook mirroring window sizes, but the default is to make windows and their content bigger.
The AVP also likes to try to improve contrast and will oversize the edges of small things like text, which makes everything look like it was printed in BOLD. While this may make things easier to read, it is not a faithful representation of what is meant to be displayed. This problem occurs both with “native” rendering (drawing a spreadsheet) as well as when displaying a bitmapped image. As humans perceive better contrasts as having higher resolution, making things bolder is another processing trick to give the impression of higher resolution.
I see different processing and artifacts happening when natively rendering on the AVP (such as when running Excel), displaying a saved bitmap from a file on the AVP, displaying a bitmap image on a web page, and mirroring the content of a MacBook. With each test image, seeing how it will display differently with each display mode is an adventure.
The sizing restrictions mostly go away when it replicates the display of a MacBook. I use a MacBook Pro M3 Pro with a 14″ 3024 x 1964 display and a ~1.54:1 aspect ratio. The aspect ratio of the mirrored MacBook display is ~1.78:1 (16:9).
Based on other reports and my observations, the AVP does different processing when natively rendering images from display lists versus displaying bitmapped images and mirroring a MacBook.
According to The Verge on mirroring a Mac:
“There is a lot of very complicated display scaling going on behind the scenes here, but the easiest way to think about it is that you’re basically getting a 27-inch Retina display, like you’d find on an iMac or Studio Display. Your Mac thinks it’s connected to a 5K display with a resolution of 5120 x 2880, and it runs macOS at a 2:1 logical resolution of 2560 x 1440, just like a 5K display. (You can pick other resolutions, but the device warns you that they’ll be lower quality.) That virtual display is then streamed as a 4K 3560 x 2880 video to the Vision Pro, where you can just make it as big as you want. The upshot of all of this is that 4K content runs at a native 4K resolution — it has all the pixels to do it, just like an iMac — but you have a grand total of 2560 x 1440 to place windows in, regardless of how big you make the Mac display in space, and you’re not seeing a pixel-perfect 5K image.”
This certainly makes sense and seems to agree with what I am seeing. It looks like the AVP first renders the image at a higher than native resolution and then scales/resamples that high resolution into 3-D space. The problem is that even if you scale up a bitmap to a much higher resolution, some detail will be lost (see Appendix on Nyquist rate).
The process appears to be different for bitmaps stored directly on the AVP as I seem to see different artifacts depending on whether the source is coming from an AVP file, a web page, or by mirroring the Macbook (I’m working on more studies of this issue).
When opening the Macbook spreadsheet in an AVP window, the default is to make the fonts about 1.6x bigger angularly. They need to be so to be roughly as readable as they are on the 14″ MacBook Pro. Combined with the wider aspect ratio, the default-sized window fills about 2.7x the horizontal field of view of the 14″ Macbook Pro at “typical typing distance” for me and is so wide that I needed to turn my head to see it all.
I can hear people ask, “So it is bigger. Is that bad if it is still readable?” It is bad in the sense that information/content density has gone down. To read the same content, the eyes will have to move more.
OptoFidelity and Gamma Scientific Optical Performance Studies
My studies use conventional camera equipment to capture what the eye sees to give a heuristic feel for how the various headsets perform. Detailed evaluations necessary for both R&D and production require specialized cameras with robots to simulate eye and head movement.
While at the AR/VR/MR conference, I met with Gamma Scientific and Optofidelity, each of whom manufactures headset testing equipment and is in the process of evaluating the Apple Vision Pro’s optical system. Optofidelity does more of a dynamic motion analysis, whereas Gamma Scientific is doing a more detailed optical study (as I understand their differences). It will be interesting to see the results of their different forms of testing.
Quoting from public statements by Gamma Scientific and OptoFidelity:
Gamma Scientific is leveraging their NED™ RoboticEye™ test platform to conduct reference optical quality measurements on the Apple Vision Pro, objectively characterizing how a user will experience the VR display. These include key performance metrics such as brightness uniformity, color uniformity, foveated contrast, qualified FOV, eyebox volume, etc. Their reporting will be critical in benchmarking the AVP against latest international standards for AR/VR display metrology.
OptoFidelity announcement: We are excited to inform you that we will comprehensively evaluate the Apple Vision Pro using the BUDDY test system. Our testing will cover a range of performance metrics for the Vision Pro, including:
- Angular Motion-to-Photon Latency
- Angular Jitter in a Stationary Position
- Angular Jitter During Movement
- Pose Repeatability (both Angular and Linear)
- See-Through Latency (Photon-to-Photon)
I plan to share Gamma Scientific and OptoFidelity results on this blog.
Optofidelity has already posted its first results in its blog article Apple Vision Pro Benchmark Test 1: See-Through Latency, Photon-to-Photon (right) and APPLE VISION PRO BENCHMARK TEST 2: Angular Motion-to-Photon Latency in VR (below) The first study confirms Apple’s claim that the AVP has less than a 12ms “photo-to-photon” delay (time from something moving to the camera displaying it) and shows that the delay is nearly four times less than the latest pass-through MR products from Meta and HTC. OptoFidelity Part 2 deals with a (head) motion to see something in the display (motion-to-photon). The Quest Pro, Quest 3, and AVP all use predictive motion for constant movement, with the AVP being particularly aggressive (lower left), but all are within a person’s ability to notice. The lower right chart uses the standard deviation for a mix of short and long movements, where prediction can be counter-productive.
What Others Think
As I prepare to post this article, we are at the two-week anniversary of the AVP being delivered to the public. We are starting to get past the “wild enthusiasm” stage, where the wonders of new technology are impressive and where people are just starting to see past the more superficial cracks, such as weight, fit, and external battery. We are getting past the “demoware” and asking, “What will this do for me on a regular basis.” That is not to say that some individuals and some applications, people won’t love it.
My analysis may be controversial, with all the “instant experts” on YouTube and social media influencers praising the AVP’s resolution and display quality. As I often say, “Anyone that has ever watched a television or slept in a Holiday Inn last night thinks they are a display expert.”
I have watched many videos and read articles on the AVP and have not seen anyone seriously discuss the color uniformity problems (I certainly may have missed someone). Snazzy Labs’ latest video is the only one I have seen that talks about antialiasing and moiré text effects and setting the images/text large by default to hide problems. Snazzy and a very few others have discussed the glare issues with the optics (which I plan to show and discuss in later articles). From a “user experience” perspective, I agree with most of The Verge’s writings and podcast comments in the last two weeks. One of my favorite quotes was originally made by Adi Robertson and cited by Nilay Patel from The Verge; “It’s magic until it’s not.”
Conclusion and My Comments
Simply put, the AVP’s display quality is good compared to almost every other VR headset but very poor compared to even a modestly priced modern computer monitor. Today’s consumer would not pay $100 for a computer monitor that looked as bad as the AVP.
As I have said before, “Apple Does Not Get different physics.” Apple can’t beat sampling theory, and even if it had 8K displays per eye, there would be some resampling problems, but fewer as it would be at the eye’s resolution limit.
For “spatial computing applications,” the AVP “cheats” by making everything bigger. But in doing so, information density is lost, making the user’s eyes and head work more to see the same amount of content, and you simply can’t see as much at once. Making everything bolder may make the text easier to read, but it reduces the faithfulness of the original image. Most of the time, the Foveated rendering “works,” but sometimes it fails spectacularly.
Am I out to break the AVP? Yes, in a way, but I am trying to be fair about it. I know for a fact it does not have the resolution necessary/desired for some applications. I’m taking a “debugger’s approach” by using my knowledge of display optics and image processing to see how the AVP works, and then I can construct test cases to show how it fails. This is more or less the approach I used back in the 1980s and 1990s when I was a CPU architect at Texas Instruments to verify our designs before the days of nearly exhaustive testing by computers, where I had to go from designing it to work to thinking “how can I make it fail.”
Per my usual practice, I have shown my results, including providing test patterns so others can verify them.
Appendix – Some More on Resampling and Nyquist Rate
There are inevitable problems resampling below the Nyquist rate that I discussed in Apple Vision Pro (Part 5A) – Why Monitor Replacement is Ridiculous. Still, they are made worse when compensating for AVP’s optical distortion and foveated eye tracking.
A simple horizontal line in the frequency domain looked like a square pulse function with infinite odd harmonics perpendicular to the line—a square dot has infinite harmonics in two directions. So even if the display has twice the resolution of the source image, there are going to be some “errors” that show up as artifacts. Then, with any movement, the errors move/change, drawing the eye to perceive the errors. Fundamentally and oversimplifying Nyquist, when rendering (resampling) a 2-D object in 3-D space, you need well more than twice the resolution of the original image to render it without significant problems most of the time. Software anti-aliasing can only reduce some of the ill effects at the expense of blurring the image. Even if the AVP had two times the pixels (~8K display), there are patterns that would be “challenging” to display.
I discussed the problem of drawing a single pixel (without getting to much into Nyquist sampling theory) in Apple Vision Pro (Part 5A) – Why Monitor Replacement is Ridiculous
Thanks for the detailed analysis, Karl.
Looking at the test pattern on the Vision Pro, with it filling up my entire field of view, I am able to easily make out four distinct lines in the test pattern. I have to reduce the size and move it far away to see the lines merge into three or two. Are you not able to make out four distinct lines in the test pattern when you look at it through the lenses?
I have the HoloLens 2 and the Quest3, and those displays are objectively worse. I agree that Apple is cheating a lot in making things bigger in order to make text more discernible, but I have been pleasantly surprised at how productive I can actually be with text-related tasks that I would never attempt on either of the other two devices, such as reading email, or editing code. I worried that the loss of information density would be much greater than it actually is. I am able to have enough code visible to be productive. And reading email or web articles is very comfortable. I am very much looking forward to an update that allows for multiple virtual Mac screens. Is it yet a complete desktop replacement? No. But it is much closer than I anticipated.
The color aberrations on the Vision Pro are so much less noticeable than on something like the HoloLens 2, that in practice, they don’t raise to the level of being distracting. The most noticeable visual artifacts to me are the scintillation that happens when there are a lot of vertical and horizontal lines being displayed, and sometimes an apparent and strange “bumpy” distortion of black-and-white text, which is hard to describe, but is almost like the text is being displayed on a bumpy piece of paper, which I think may be due to the foveated rendering.
Any hints on how to trick the Apple Vision Pro into displaying the eye chart at the proper size and distance?
I made the test pattern big enough so I could see all four corners at the same time with both eyes. The pattern appeared very large in my FOV. To take the picture I then recentered to take the picture through the left eye’s display (see the picture). At this size on the AVP, the test pattern is about 2x bigger than on my 28″ computer monitor (which being a 4k can display two of these test patterns side by side).
To see 4 (somewhat blurry) lines I have to move in close enough were only about 4 of the 5 vertical panels fit in the FOV and the top and bottom of the test pattern rounded off by the optics.
Being better than Hololens 2 is damming with faint praise. The Hololens 2 has to be about the worst display I have ever seen for color uniformity.
For the Snellen Chart, I started with the SVG file on Wikipedia and then output it as a high resolution bitmap (PNG file) in Photoshop. I then displayed it on a MacBook Pro and resized the window until the chart was the same size and from the same distance is one on my 4K 28″ monitor.
In going back and looking at the color uniformity, the cyan outer rings cannot be missed and seem to take up about 5-10% of the FOV. The reddish tint in the center, however, is less noticeable if both eyes are open as if the color variations are different, they tend to cancel each other out.
I went back tonight looking at the color uniformity issue. To be clear, the AVP white uniformity is not horrible, but it is not great either. Tonight I did a back and fort comparison to the Meta Quest 3.
The Quest 3’s white uniformity was much better (hard to see anything wrong with it). The AVP uniformity is definitely lacking by comparison. Perhaps a bigger problems is that as you move your head and eyes, the color moves and changes (I assume with eye tracking) which makes the color stand out more. I also liked the color of the human pictures better in the MQ3 on a white background (KGOnTech 1920 by 1080 Tuff Test Black on White). The AVP appeared to make the person’s skin tones a bit cyan (lack of red).
As some are pointing out, the AVP looses a LOT of contrast when there is a lot of white content. The MQ3 optics seem to be vastly better with dealing with and preserving contrast with a lot of white in the image. Since with pancake optics, all the light is bouncing back and forth off of several surfaces it is famously prone to ghosts and contrast loss. I would like to see an ANSII checkboard contrast test on both devices (hopefully someone like Gamma Scientific will test and give some results. I’m not picking sides here, but just reporting on what I am seeing.
is it objectively better, yes
it is good enough for productivity – not for the type of text heavy productivity i do – words that fill the screen (not coding which is way less dense than email or word documented)
so for me this is ‘quest 3 is meh’ for text and ‘avp is less meh’ for text, meh is till meh
if you can cope with blurry text, great, glad it works for you, it didn’t for me, and the mac monitor remoting was one of the most disappointing features for me on mine.
Disclaimer: I’m no display expert, just your average Joe chiming in, so take my opinion for what it’s worth.
I think what’s often missed in these comparisons is that the Q3’s virtual monitor quality isn’t solely determined by the headset itself. For example, I tried the virtual monitor on the Q3 using the Immersed app with three different laptops, and guess what? The clarity and quality varied big time. One had an i7-1165G7 with Iris Xe graphics, another a Xeon 1535m v6 with a GTX 1080, and the third was a MacBook Pro with an M3 Max (16-core CPU, 40-core GPU). I’m not tech-savvy enough to understand why the quality was different (maybe someone can tell me), but each combo gave me a different viewing experience. Surprisingly, the Q3 paired with the MacBook Pro offered a more vibrant and crisp display compared to the AVP. Plus, it had less blur and a better field of view (FoV), making it a winner over the AVP in my book for a monitor replacement.
Anyway, my point is that stating something like “the Quest 3 was objectively worse” doesn’t mean a whole lot without further context because the whole setup matters (machine, cpu, gpu, etc). If you used the Q3 as a monitor replacement with my Lenovo, then compared it with my MacBook Pro, the difference is night and day, so comparing either against the AVP would yield different results.
I thought everyone that watched a TV or slept in a Holiday Inn Express last night thought they were and expert 😁.
I think a lot of variations you see are due to different amounts of rescaling/resampling. What resolution the virtual monitor is trying to emulate and the quality of the scaling software can have as big an effect as the resolution of the display.
Every pixel in the source display is being mapped to at least 4 pixels the display space (as I discussed here: https://kguttag.com/2023/08/05/apple-vision-pro-part-5a-why-monitor-replacement-is-ridiculous/#rendering-a-dot). Even if the headset’s display was identical or higher resolution, you have a remapping into the 3-D space to make the virtual monitor look stationary. I don’t know the exact process for every application, but one method is to first scale the image into a higher resolution space (The Verge reported that the AVP remaps to 5K resolution) and then 3-D maps the intermediate higher resolution into the lower resolution 3-D space ALONG with distortion and color correction which is dependent on eye tracking of the AVP (it is theoretically possible that they do the remapping in a single step and that what the Verge reported was a simplified conceptual explanations. Regardless and to kept it simple, each pixel in the source is going to have to cover at least 4 pixels (and likely more than that) pixels in the display.
The process also can be a bit different for “native” applications where the pixel scaling can be calculated on the fly. With the AVP, it still “feels like” (from what I see) that they may be rendering to an intermediate resolution for office-type applications.
I’m doing some studies right now comparing the AVP and Quest 3 “optics” and in particular the glare issues with the AVP that are being widely reported. The Quest 3 appears to be better in this regard with a mostly white/bright background. The AVP’s optics seem to have a lot of problems dealing with bright screens compared to the Quest 3 and a lot of the light bounces around causing glare (a loss of contrast). While they are both “pancake optics” the designs are very different with the AVP using a concaved lens (the Quest 3 is convex) near the eye (which may be causing reflection issues with the Zeiss inserts) and a curved polarizing mirror and quarter waveplate (they are flat in the Quest 3).
The color uniformity seems better with a white screen on the Quests 3 compared to the AVP and skin tones appear better with the Quest 3, at least against a white background.
The raw resolution of the AVP is clearly better than the Quest 3, but it still only gives about 20/30 vision with “native” content and the display resolution is not the only factor. Yes, you can’t see the pixels, but part of that could be due to the optics “softening” the pixels a little.
@Patrick, I can also see a bubble distortion you are talking about. It is indeed more visible on text, but I can see it elsewhere (e.g. on images) too. For me, it’s only noticeable when you start moving your head around. Looks like there is some slight distortion in the lens (left one for me). It is my second unit, and I don’t recall if I’ve seen something similar with my first one. There were definitely some weird artifacts there as well, including a pretty big blurry area in the left lens. The second unit has much smaller blurry spot, mostly close to the edge of the lens, but it has this bumpy distortion that looks like someone is holding a magnifying glass. The effect is very small though, but I can’t unsee it.
Great technical content. I love seeing passion and precision in technical discussion. It’s also important to acknowledge that technical truths don’t necessarily correlate to utility truths. There may be a better way of saying this, but an example is that technically a photo shot with a modern DSLR or mirrorless camera is better than that of an iPhone… the iPhone in your pocket takes better photos that the DSLR or mirrorless camera you don’t own. Simply, people who are “replacing monitors” with Apple Vision Pro (like myself) aren’t doing it because it’s better than my monitor, but because it allows me to take my monitors and computer with me – in spite of the technical inadequacies. A 300dpi laser printer in 1989 wasn’t better than plates and ink in the same time period, but it sure reduced the time to eyeballs for printed content.
The question is, whether people will take their AVP with them instead of a Laptop or with it. A laptop is much more conveniently carried as it folds flat. If want a second monitor for say videos or unrelated content, I can use at tablet or my cell phone, once again the two devices combined are easier to carry than an AVP.
For doing work, I would much prefer looking at a laptop screen. You milage and preferences may be different. I know there are some VR and immersive focused people that would prefer a bigger and more immersive display. There may even be some tasks where the AVP might be better; I would like to see some time and motion studies as well as studies of any issues with long terms “exposure” to using the AVP. For example the fact that the cameras and view from the AVP is not lined up with the eyes is likely to cause problem over time as the visual system adapts to the misalignment.
If there are any specific questions you have, I’m happy to help. Since launch day I’ve been in the headset for an average now of 7 hours per day. I take both the headset and my 16” M1 Max MacBook Pro with me every time I leave the house to work out in city (Dallas, Texas). From coffee shops to hotel lobbies, even public co-working spots. I also take a dedicated Magic Keyboard and Trackpad so I can type and navigate without the laptop being open. It gracefully assumes control of the laptop when it is connected giving me access to that world as well. Again, happy to help! The MacBook Pro now has a specific use case (debugging websites with the inspector) since web inspectors aren’t available in *OS 🙂 Here’s link to my “work” setup out in the wild: https://www.instagram.com/p/C3YqDuePdKQ/
Thanks for the feedback, I would like to understand your use model more. I will also be interested to see if you keep it up.
I don’t understand out putting on a headset and using an external keyboard and mouse is faster and easier than simply opening a laptop.
Also be careful when you are out and about as it isolates you more than you might think. What are you doing about theft insurance. I’m hesitant to take mine on trips due to the lack of good insurance options.
i wonder what the effects are to one’s vision or even neural circuitry when looking at a VR display for extended periods. additionally, there’s the occlusion of peripheral vision which in some studies showed it negatively impacted reaction times after the headset was taken off.
It would be good to have some studies done on the effects. If possible, it would be good to get pointers to studies that have already been done.
Karl, my use case is similar to Michael’s. Using AVP instead of MacBook’s own screen has a few advantages to me:
1. I can get much higher pixel resolution and pack more contents on one screen than with my 14” MacBook while keeping everything perfectly readable. Yes, it’s not as sharp as Retina Display on my Mac, but it’s not terrible. After all, we used pre-retina displays just fine for many years.
2. The posture is much better – I’m not sitting hunched over my laptop, especially when it is on my lap.
3. When traveling or working in public places, PRIVACY! This is a big one, as I can finally work on sensitive documents without risking anyone to look over my shoulder.
4. While it allows to mirror only a single Mac screen, you can augment it with multiple native windows – it’s like having multiple iPads around you.
5. I like to pace around when on Zoom/Teams meetings, so I can blow up the window to cover the entire wall, and can see their screen share without being chained to my desk.
6. And this is probably unique to my eyesight, but I have significantly less eyestrain as I can work on a giant screen at a comfortable distance vs. sitting close to my 32” 4K display or worse, my built-in MacBook Pro. I usually start having double vision after working for 30 minutes or more, unless I blow up the fonts so I could read the display from ~4 feet away. With AVP, I can spend a couple of hours working, and my eyes still feel good. I get a bit more of a dry eye when wearing AVP though, but it beats seeing double for sure!
Having said that, I wouldn’t mind having higher resolution so everything is sharper.
However, the biggest issue I’m having with this headset is glare/reflections. I wonder why some people claim that they don’t see them – I tried another unit at the store, and it was the same. It is somewhat manageable in a brightly lit room/virtual environment, but gets pretty bad with dark backgrounds. I still have some time until my 14 days run out, but if I return the headset, it would be the reason. What’s interesting, the glare seems to be worse with “deeper” light seals, like my 25W. If I take the seal off and bring my eyes closer to the lens, the FOV expands and reflections/glare becomes a bit less noticeable.
@ Michael Sitarzewski
I agree. The analogy that comes to my mind is this: a touch screen is objectively less accurate than using a mouse. However, a touchscreen gains expressiveness (multitouch gestures), directness and portability. This makes it a superior interface to a mouse for some applications, such as controlling an audio mixing desk, or interacting with a pocket device. Understanding the touchscreen’s limitations (its lack of accuracy) by objective measurements and extensive testing makes for better touchscreen applications (i.e, knowing what size to make GUI elements, or making a choice to not sell a MacBook with a touchscreen).
Sadly this evidence-based approach isn’t universal – which is why we have cars that use touchscreens for functions that should be controlled with, for example, a rotary knob. A driver can reach for a physical knob to control the air temperature whilst keeping their eyes on the road ahead.
What the AVP loses in resolution and colour accuracy might be made up for by allowing the user to use their body more. The brain tends to offload cognitive tasks onto the body if it can –
the concept of placing windows and documents around sounds appealing, I look forward to it being tested. Being able to stand up and adopt different postures brings health benefits over sitting at a monitor. Interesting times.
Hey Karl,
Thanks for the detailed analysis. Did you calculate the FOV of the pancake lens inside AVP?
This slide gives a FOV of 97° x 74°: https://i0.wp.com/kguttag.com/wp-content/uploads/2024/02/2024-02-09-10-8423-min-and-max-fov-copy.jpg?ssl=1 (you can jump to it by searching this article for “FOV increases”)
That slide also says “1.46x bigger FoV” with the camera simulating a very close eye position, which looks to be specifically referring to horizontal FOV, vertical FOV is increasing by a more modest 1.11x (based on measuring the slide), unless the top of the vertical FOV is out of frame? Roughly 146° x 82° by my estimate (again, using the slide), which may only be possible w/o prescription lenses & for some face shapes?
Brad Lynch made a measurement using wimfov, which is unique to his face + light seal, hopefully others will do so as well:
https://www.reddit.com/r/VisionPro/comments/1aqiljz/fov_measured_with_wimfov_by_brad_lynch/
Thanks for the reference to Brad’s estimates. A few things:
The vertical FOV in the picture close to the lens picture is significantly cut off by the camera lens. You will see with the red dotted line where I estimated the cutoff horizontally but did not do the same vertically (the camera has to be off center to capture the picture correctly as FOV is off center in each eye. I would assume the FOV difference is similar in both directions.
I didn’t see directly in Brad’s number the “binocular overlap” or how much of the FOV is shared by both eyes. If I add Brad’s left eye’s left of ~56 degrees to his right eye’s right of ~56 I get 112. Adding the 98 degrees of both eyes gives 196 suggesting a binocular overlap of about 84 degrees. This makes sense and in the range of the Meta Quest Pros ~80 degrees of binocular overlap (https://risa2000.github.io/hmdgdb/hmd_cfgs/MetaQuestPro_Native_R72.html).
The “average PPD” does not tell the whole story. It is important to also know the PPD in roughly the center 30 degrees of the FOV. The eye will saccade (jitter) but typically (it varies from person to person) likes to stay within 30 degrees and thus the only area that can remain sharp for long periods of time. This is why I reported the “peak” FOV. It is somewhat legitimate to say that most VR optics have a “foveated optics” with denser pixel spacing which are also sharper in the center.
Re: overlap: Interestingly wimfov calls it “67%” (bottom left in images from https://twitter.com/SadlyItsBradley/status/1757501944686408066), which I guess is the percentage of overlapping area? Haven’t tried measuring that.
I measured overlap from +30° to -30° using Brad’s wimfov chart (by measuring pixels), you can see those numbers here: https://www.reddit.com/r/VisionPro/comments/1aqiljz/comment/kr8pupb/
Is there any standard way of calculating overlap when each eye’s view is rounded? Standard description of HMD binocular overlap just deals with “rectangular eye view” case. Not sure what hmdq (https://risa2000.github.io/hmdgdb/) is reporting, probably overlap at 0°.
Re: avg. vs. peak PPD: Indeed! A distinction whose importance I owe in large part to your prior articles. I’ve been playing with average PPD numbers primary because it was the only thing I could come up with estimates for, while waiting for you to work your magic!
On that topic, is there any standard way of calculating peak PPD? Reporting the average over the central 30° area (circle with 30° radius) vs. the central 10° area may yield different numbers, though probably not markedly so in most cases, as manufacturers presumably want to keep distortion low in this area.
I had really really bad display color calibration issues so much that i had to exchange my units for reds in left and right eye being extremely off but no one else seems to have talked about this. My new unit while much better is still not a perfect match in red hues which can be tested using the color chart in the color filters accessibility menu
Thank, but I can’t find any directions as to how to use the feature. I found under accessibility shortcuts where I can enable the color chart with a triple crown click, but then I don’t see anything happen when I triple click.
Could you provide what I need to click through to enable the feature and what I should see?
Very interesting review. Will try to demo AVP when it comes to London.
I’m currently using Tobii eye tracking with dynamic foveated rendering and Almalence digital lens software with my Pimax Crystal; its noticeably improved the perceived resolution /image sharpness whilst reducing pupil swim and chromatic aberration.
For me I have a very similar usage pattern to Eugene’s post earlier – I mirror my Macbook screen, and have several different Vision Pro apps around (like Slack and Safari).
I find this very handy compared to switching between apps on the desktop only. Especially for having content like music or videos play to the side while I work, because I don’t have to go hunting though desktop windows to find them to adjust what they are doing, as the float right to the side. To me having a number of apps spread around a primary work monitor is actually a bit more useful than two monitors for most use cases.
I do mostly coding or other document related work, and for me I find the display really good. Yes not as sharp as the main Macbook display, but also making it “larger” is easier for me to read, whereas sometimes on the Macbook I would use Accessibility Zoom to zoom in the screen at times.
I also like as Eugene mentioned, having a laptop screen that is positioned higher for better posture. And a huge bonus coming up is, I can work outdoors (like on a porch) in conditions that would wash out a laptop screen.
I just find the utility of it so much greater than any loss of visual quality…
I do notice light blooming issues at times but so far really just in movies. I do find it a little annoying at times, but again the utility of being able to watch a movie on an apparently large screen in any comfortable place I can find, is kind of invaluable.
I do enjoy reading your deep analysis of the optics, thank you for your hard work on this! I look forward to hearing more.
Thanks, I appreciate your input and relating your experience.
It looks like there is a great variability in quality between the units or even individual lenses/screens. For instance, my unit has a blurry spot on the right side of my left lens, covering about 1/5-1/6 of FOV horizontally. It’s similar to what is described in this post: https://forums.macrumors.com/threads/ive-been-using-a-defective-vision-pro-for-2-weeks-replacement-is-night-and-day-optically.2419690/
The right one is sharp pretty much edge-to-edge. However, the left one has better color uniformity, and the right one has more fringing on the edges.
I don’t mind minor color uniformity issues, at least for my use, but the blurry spot is a bigger deal. I dismissed it at first, thinking that it could be a bug in foveated rendering, but given that the right one is not nearly as bad, I’m going to either get a replacement unit or return this one and wait for Gen 2. I’ll see if their demo units have similar issues.
Hi Karl, thanks a lot for the detailed analysis! Forgive me for being someone slept in a Holiday Inn last night 🙂 I just want to point out that the distortion in the optics might be purposefully designed. Sometimes for laser scanning objective, negative distortion is added on purpose to create the so-called f-theta lens (as opposed to f-tan(theta) in regular optics), which has a constant angular resolution. I suspect that Apple and Meta are both trying to make an optics that has such constant angular resolution which might be beneficial for VR/XR as it better reproduce virtual 3D objects. Contant angular resolution and constant spatial resolution (2D) are intrinsically incompatible so surely it will never be able to reproduce a flat TV/monitor perfectly. It might be able to reproduce a “spherical” screen with contant pitch though… In any case, one can always argue that it’s a good thing to have more resolution (even spatially) in the center.
No real disagreement. I think that severe pincushion distortion is a natural consequence of trying to create an extremely wide angle lens to create a wide FOV with about a 2 meter apparent focusing distance. I think all or almost all VR have this distortion. I also agree from a a techno-marketing perspective it does have “foveated optics” effect with the center resolution being higher.
[…] More info (Karl Guttag’s analysis of the Apple Vision Pro display) More info (Optofidelity on the latency of the Vision Pro passthrough) […]
Thanks you Karl. Very good write up, and appreciate your technical approach regardless if it puts the product in good or bad light. I look forward to further analysis.
Thanks
Interesting writeup as always!
How are you keeping the AVP displays turned on when using a camera to shoot through the lens? And how are you ‘aiming’ the eye-tracking without having an eye in there?
Ben, it can be surprising easy once you know the trick. I should point out that this trick was found by iFixit. If you get the headset “running” and cover one eye (helps to have the light shield removed), the headset will keep running when you take it off. If any any point the cover is remove and both eyes are “exposed,” the headset will shut off. I have found through (painful accidental experiment when it stopped working) that is is better to cover with a white microfiber than a black light absorbing material. Usually, I tape the cloth in place to keep it from accidentally uncovering the lens. I understand there are some other ways that I will be exploring.
Thanks for the tip! 1) What app do you use to display images in AVP? 2) When you take off the headset, how do you aim the virtual target image with the external camera? By maintaining the headset to the same direction/orientation?
1) I use several different applications. I have used the web browser to look at images on my website, I have used apps running on the AVP (Excel and picture display), and I have used connection to a MacBook Pro M3 Pro.
2) I wear the headset to get things generally arranged and lined up. I then have a clamp to hold it on a Tripod that is roughly at my head position. I then do some fine tuning with the tripod adjustments once the headset is clamped on the tripod.
I was not able to see the red color non uniformity in mine, but I can see the cyan in the border if all the display is white, but to be honest, they hide it somehow in any other image I can only see it when it is full white.
I think I do like it better for on-the-go laptop monitor replacement than you seem to like it. It will depend on the task, but in addition to a bigger display the ergonomics are a bit better (looking ahead rather than down on my laptop), but I still like physical monitors better. I can see some display artifacts in strange images with some CAD tools, but overall seems a worth trade-off if you are not in your desk.
Apple TV+ Immersive content, while there are only a few, is absolutely mind blowing. You do feel like you are there.
I do seem to find it more comfortable than most reviewers. I think that people over tighten the Solo strap, and that they also put it too low, in the back of the head instead of a little higher in the back of the crown.
The red lack of uniformity is definitely there, but the camera does exaggerate it a bit. Using both eyes also helps as color a person sees is more or less the average of both eyes, whereas with resolution, both eyes can see better than either eye, with color they tend to average. Try looking at a white screen and moving your head around, you should see the pink area move. Note that the AVP’s eye tracking appears to be constantly trying to compensate for the color problems. It is most dramatic to me when I switch back and forth from the AVP to the MQ3 as the MQ3 looks so much more purely white across the screen.
BTW, in addition to the AVP’s optics having poorer color uniformity, they are appear (preliminary result) to be softer/blurrier than the MQ3’s optics. Results, to be published.
The “information density” with the AVP is considerably lower than you get with a monitor. This means you eye and head will have to move more to take in high resolution information.
Ergonomically you don’t want to look straight out at a monitor, you want to be looking down (search for monitor ergonomics). The common ergonomic recommendation is to have the top of a monitor at or below your sightline, so you are always looking down. Everything in human vision and factors (including how your neck works) is bias to looking down. The saying goes, that for the caveman, almost all predators that could kill them was on the ground and not in the air.
Your articles never fail to captivate me. Each one is a testament to your expertise and dedication to your craft. Thank you for sharing your wisdom with the world.