Introduction

In my last article, Canon R5 Mk ii Drops Pixel Shift High Res. Is Canon Missing the AI Big Picture? I showed how IBIS High Resolution was useful in improving resolution and reducing moiré/aliasing. Having received my R5 mk. ii (R5m2), I set up a simple resolution and moiré test to compare the R5m2 with its “AI” In-Camera Upscaling to the original R5 with IBIS High Res (IBIS HR).

Having now used Canon’s in-camera upscaling, it was worse than I thought. The upscaling ONLY works on JPEG and HEIF files and, thus, has thrown away the information from the RAW pixel sensors. It won’t even work with RAW files. Losing the info from the RAW file makes the feature even worse. I will be comparing the R5m2’s in-camera upscaling versus Adobe Photoshop’s “Super Resolution” from the RAW file of the same picture.

I have had some feedback that people think I am just worried about resolution and my particular use in testing AR and VR headsets. However, I think it goes to a bigger issue of what a camera can do to create better images. IBIS Pixel Shift (Canon term: IBIS High Res) creates more real information, whereas “AI” upscaling generates fake information that may look eye-pleasing but can also worsen artifacts. I have some background in image processing, having led the technical development of the TMS320C80 Multimedia Video Processor (MVP) from 1989 through 1996. I think about “information theory” from having designed graphics and image processors for about two decades at TI.

I have been shooting with Canon DLSR and mirrorless cameras (R5 and now R5m2) since I bought Canon’s first DSLR, the D30, in 2000. The R5m2 has some amazing features, particularly advanced focusing capabilities, so this is not a criticism of the camera broadly. However, from my technical perspective, Canon seems to have gone in the wrong direction, scrapping IBIS High Res rather than improving it (by simply NOT processing it in camera and saving RAW images). I think Canon compounded their mistake by adding (as will be shown) very poor in-camera scaling from JPEG and HEIF files.

The R5m2’s Upscaling is Worse than I Thought – It only works with JPEG and HEIF files.

Beyond the fact that upscaling inside a camera will be worse than what can be done with more processing power outside the camera, the R5m2 upscaling only works with JPEG and HEIF files. Starting from already “de-Bayered” rather than RAW images throws away details/information from the color photosites that could help upscale (some products running on PCs, including Photoshop’s “Super Resolution,” require RAW images).

Canon R5m2’s on-camera software seems completely backward. If you shoot RAW/cRAW, you must convert it to JPEG or HEIC before upscaling, which adds artifacts and loses detail in the source image. Trying to find images on the little screen and going through its menus on the back of the camera is like going back 30+ years of human interfaces. I use two large monitors at home to sort, cull, and pick photos I want to edit. The experience left me wondering, “Who on earth thought this was a good idea?”

After converting my RAW images to JPEG and HEIF, I had to wait while the tiny processor with a small amount of memory upscaled my images. At the same time, I sat looking at my desktop computer, which has 16 CPU cores and an Nvidia graphics processor, which my other photo editing software uses.

Below is a comparison between taking a RAW picture on the R5m2 converted to HEIF (in-camera) and then upscaled (in-camera) to taking the same RAW picture and directly upscaling the RAW file in Adobe’s Super Resolution. The Appendix discusses the picture and some other comparisons I ran.

Canon’s IBIS High Res Needed Improvement, Not Removal

The WAY Canon implemented IBIS High Res (IBIS HR) on the R5 is very limiting. My through-the-optics pictures of AR/VR displays could live with the limitations of IBIS HR. I typically have to shoot at slow shutter speeds to reduce/eliminate temporal artifacts that would not be visible to a human eye. The IBIS HR solves a moiré problem and sometimes improves resolution, which is not unique to my use in photography AR and VR headset displays. Most of the time, the R5’s 45 megapixels (8192 by 5464) are more than is needed, but it is great to have another “gear” you can go to when necessary.

The “more normal” area where I will use IBIS HR is to make a panorama. Typically, a “panorama” takes multiple pictures and pans (either on a tripod or by hand), with about 50% overlap, the camera between them. Later, the software will stitch the pictures together into a single image. With IBIS HR, you could take a wider shot to get about the same level of detail as three shots in the panorama.

The R5 IBIS HR required the camera and everything in the subject to be stationary for the time the R5 took to take nine pictures. If a leaf moved on a tree, it would become a digital mess as the on-camera firmware was not that “smart,” and the image was saved as an immutable single JPEG image rather than simply saving the RAW files. The camera should have saved the nine RAW pictures (or optionally more to help with motion) so that smarter software on a computer could combine the images and deal with motion. As a user, I feel like Canon wants to control everything, even when it will give worse/unusable results, and it would be easier to give the user access to the raw files.

Canon R5 IBIS HR vs. R5m2 Resolution & Moiré Shootout

I set up a simple shootout between the R5 with IBIS HR and the R5m2’s AI upscaling mode. For this test, I printed out a series of vertical lines that are 0.3mm apart on 0.3mm spacing and then on 0.5mm and 1mm. The test pattern was angled at about 45 degrees so the lines would get progressively closer, as the camera saw. There is a focusing area about 1/3rd of the way into the pattern (since focus falloff is asymmetrical). I shot at 16mm, f11, from approximately 1 meter away to give a large DOF and to make the test pattern just beyond the limits of the R5/R5m2 native resolution (even with a small CoC, everything should be considered in focus). All pictures had the same 16mmf2.8 lens at ISO100 and 0.5-second shutter speed. The test pattern ended up being a small crop in the center of each image. I swapped cameras on a tripod and moved the lens between cameras. Even though the shooting settings were IDENTICAL with the same lens, the R5m2 consistently seemed to be about 1/2 stop darker than the R5.

The picture below shows the full test pattern and the setup. It is a center crop with no scaling (click on it to see it without any scaling), showing the test pattern taken with the R5’s IBIS HR without any other scaling. As IBIS HR increases the image size linearly by 3X, whereas the R5m2 scales the image linearly by 2X, the R5’s IBIS HR image will be scaled by 66.67% in the subsequent comparison. While the paper was taped down to a computer monitor’s glass, it was still a little wavey on the right side.

I will also crop more of the rest of the pictures so they will not be scaled when this article is viewed at 100% on a computer monitor (and thus not introduce new moiré issues without clicking through to the full image).

Below is an R5m2 picture of the test pattern originally shot in RAW and converted in-camera to HEIF in the R5m2. It was then scaled by 2x with Photoshop Bicubic to match the resolution of the R5m2’s in-camera upscaling. You should note the color moiré in the 0.3mm lines on the upper left, the intensity moiré patterns in the 0.5 lines and some color moiré on the far right, and a mix of color moiré on the left and the intensity moiré on the right of the 1mm lines.

The image below was taken with an R5 using the same camera setting, lens, and distance from the test pattern with IBIS High Res and then scaled down with Photoshop Bicubic to match the R5m2’s in-camera upscaling size. Note that the 0.3mm lines are visible for the over-have of the test pattern, demonstrating increased resolution. No substantial moiré is visible in this test setup, but moiré can still happen even with pixel shifting if the subject has a pattern(s) at the right frequency.

Next, we have the R5m2 with its in-camera upscaling of the (unscaled) JPEG image. Regarding moiré (both in color and intensity) and resolution, it does not look much different than the source image scaled by 2x without “AI.”

Because the R5m2 can scale HEIF and JPEG images, I tried converting the RAW to both JPEG and HEIF and scaling them in-camera. The upscaling from HEIF is shown below. There are minor differences with some more compression-like artifacts in the JPEG. Both the HEIF and JPG scaled versions create a lot of artifacts. Particularly note the area around the “focusing rectangle” in the center with a cross in it.

I also want to see how the R5m2’s in-camera upscaling compares to an upscaling from a raw file. The picture below uses Photoshop’s Adobe Camera Raw’s Super Resolution to upscale the same raw file used to create the JPEG and HEIF files for the R5m2’s in-camera upscaling.

R5m2 In-Camera vs. Photoshop RAW vs. IBIS HR Scaling

The previous images showing a larger region show the moiré effect. Cropping down and zooming in will show some of the artifacts side by side. The cropping of the R5m2 In-Camera upScaling (I-C-S), Adobe “Super Resolution, and R5 IBIS HR, plus crops with 2x pixel replicated zooms of the rectangle I used to aim and focus the camera. The comparison of the R5m2 scaling to the Adobe Super Resolution demonstrates the image quality issues starting from the non-RAW file on the R5m2 and the lack of processing resources on a camera. These problems are similar to the picture of my eye shown earlier.

Computation Photography With Stand-Alone Cameras

While computational photography is well integrated into higher-end smartphones for everything from night shooting to extended dynamic range and panorama shooting, stand-alone cameras, at best, have taken a scattershot approach, adding some support for some operations but not others. There seems to be no strategy or consistency in supporting computational photography.

While I recognize that advanced focusing and lens/camera stabilization are core features for modern cameras, it wouldn’t take much for stand-alone cameras to better support computational photography. Much can be done by saving multiple RAW images and tagging them in the metadata as a group (some cameras do this).

In the “Pixel Shift” (IBIS HR) case, the function cannot be duplicated unless the camera controls the IBIS movement. If JPEGs are stored rather than RAW files, much information that could be used to deal with possible motion artifacts is lost. When implementing these features on a camera, camera makers should ask themselves whether the implementation is better-performed in-camera. In the case of R5’s IBIS HR, Canon has gone from implementing it poorly with a JPEG output to eliminating it and adding a terrible method of in-camera upscaling.

Conclusion

Canon’s R5 mk ii In-Camera Upscaling is a poorly executed gimmick feature. A computer using the RAW files can produce much better results. The fact that R5m2 can only upscale HEIF or JPEG files, and not RAW files, shows that it was doomed from the start.

Unfortunately, the In-Camera Upscaling it being considered a replacement for the IBIS HR (pixel shift) mode on the R5 that was eliminated. As demonstrated, even scaling on a computer from the RAW file cannot match the resolution improvement from IBIS HR. I can only hope that the removal of IBIS HR is only temporary, as it was added to the original R5 in a firmware upgrade.

If Canon adds back in IBIS HR on the R5 mk ii, I hope they save RAW files rather than doing the on-camera processing and saving a single JPEG as they did on the R5. A vastly better job could be done, including dealing with motions, with superior processing, memory resources, and the ability to run different algorithms.

When done well, computational photography combines information from multiple images to produce a single image, and it leverages additional information rather than just guessing/predicting. Canon would have been better off improving IBIS High Res, and by that, I mean doing LESS and saving RAW pixel-shifted images.

Appendix: More on the Photo of the Eye Scaling

The classic area that people test for resolution is in a picture of the eye area with the detail in the iris and eyelashes. So, I quickly set up a (self) headshot with the R5m2 eye focusing and tried different scaling methods. The image on the right shows the full-frame picture that has been greatly scaled down, with the background blurred to give an idea of the size of the eye.

The matrix below shows various scaling methods. I have scaled everything to the same size. The Original picture was taken with RAW. I covered the RAW to HEIF with the HEIF quality set to 10 (best). Four areas stand out: the eyebrows, eyelashes, IRIS, and blood vessels. The description of the rows is as follows:

1. For the HEIF, I have enlarged it by pixel replication (to make the pixels bigger) and using Photoshop Bicubic scaling. You can see how the eyebrows are both lost and take on a pixelated/jaggy effect. The IRIS has lost resolution, and some of the eyelashes are all but gone. 
2. This row has the R2m2 in-camera upscale from the HEIF and a simple bicubic enlargement followed by an Unsharp Mask with a radius of 1.5 at 100%. I find it very difficult to tell the difference between the two pictures.
   1. Not only is all the detail in the eyebrows lost, but there are some jaggy artifacts.
   2. The eyelashes are blotchy, and some of all but missing
   3. There is a lack of detail in the iris
   4. The blood vessels in the eye are all but gne
3. This row shows Photoshop Super Resolution from the original RAW file (ACR “enhance Super Resolution” works on the R5m2 CR3 files).
   1. There is a dramatic improvement in the eyelashes over the R5m2 in-camera upscale
   2. There is more detail in the eyelashes
   3. There is detail in the ISIR not seen in the in-camera upscaling
   4. Adobe Camera RAW is noisy, and the USM sharpening enhances that noise.