Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
AWE 2021 Part 1 – Tilt-5 Was Magical
Introduction
It was nice to get out and actually look through AR and VR displays at AWE 2021. I started to write about everything I saw at AWE, but to me Tilt-5 was such a standout that I decided to start with an article about my Tilt -5 experience and later follow up with several articles on other things I saw.
I enjoyed Tilt-5 the most of everything I saw at AWE. It was surprising because I had known of Tilt-5 and its predecessor castAR for many years but had not seen it. Unfortunately, I had dismissed its use of a retro-reflective screen as a desperate act and hokey, and with castAR failing in 2017, I figured that was it. While I admired CEO Jeri Ellsworth’s tenacity, I wondered she was a bit of a Don Quixote with a loyal following.
But then I finally got to see the Tilt-5, and to me, it was instantly magical, and I personally ordered a unit the same day. I have seen a lot of AR and VR technology, and I don’t get this feeling too often. Tilt-5 is not the be-all and end-all of AR, nor are they trying to be, but it seems to be good at what Tilt-5 is trying to accomplish, a game system. I’m left in this article to start to explain why it felt so magical.
Tilt-5 Experience
In walking, the exhibit floor of AWE was tucked near the back, and I rounded a corner to find a crowd of people around a small booth. I saw the game board and realized that this must be Tilt-5 before I saw the sign (below left is the first view I saw). Before I got to try it on, people were giggling and going, “oh wow,” and then I put the headset on, and I was joining them (I would suggest watching Norman Chan’s Tested Hands-On with Tilt-5 for a similar, but much longer, reaction).
Tilt-5 turns the AR concepTilt-5 turns the AR concept a bit inside-out using a retro-reflective (glass beaded) screen. What Tilt-5’s CEO Jeri Ellsworth calls “AR Somewhere.” I’m told all the time you have to “see it to believe it,” and 99 out of 100 times, the experience is underwhelming. But with Tilt-5, you really do need to see it. Consider the following:
- The 3-D effect is perhaps the best I have ever seen; virtual objects appear above and below the surface.
- Real-time 3-D image warping in the glasses (using and Intel Movidius processor) shows no perceptible delay between head movement and the image responding. This processing also reduces the burden on the host (PC, tablet, or cell phone) generating images.
- Real-world occlusion of the virtual just works (something that no one solves well with normal AR glasses)
- It (to some degree) addresses vergence-accommodation conflict (VAC) (see Tilt-5’s demo video on VAC here – more on this later)
- It has a 110-degree field of view – by far larger than any other optical AR headset
- Virtual images lock well to the real world – thanks to the locator dots on the board
- Plenty of eye relief for people with glasses
- Priced at $359 for the XE model (a $299 entry model to come later)
What makes it “magical” is that everything seems to work the way it should in a way I have not seen in any other AR device. Below is a through one lens picture (a set of short through the (one) lens video clips is here). Of course, the single picture below cannot convey the feeling of binocular stereo depth. The retro-reflective game board appears to be a portal into another world.
Below is a series of short video clips shot through the lens taken by Tilt-5. The moving camera helps convey somewhat the 3-D effect (better seen with both eyes):
Tilt-5 was demonstrating at AWE that the virtual work could be extended beyond the Tilt-5 demonstrated at AWE that the virtual work could be extended beyond the gameboard provided some tracking markers were in view (see right). Retro-reflective fabric is relatively inexpensive (about $10/square yard on Amazon), making for expanding the Tilt-5’s virtual world.
Other Features and Technologies
Other Features and Technologies
This blog primarily concentrates on displays and optics, not the other aspect of AR/VR/MR products. But for completeness, I would like to mention a few other features and technologies in the Tilt-5.
The Tilt-5 comes with a control stick wand with 6DOF tracking based on IR tracking and inertial sensors, and it also has buttons that connect to the glasses via Bluetooth.
The headset includes dual speakers, and a microphone, and inertial measurement. The most interesting processing hardware is an Intel Movidius vision processor for real-time 3-D image warping locally in the headset. This headset supports quick 3-D motion feedback with head movements without waiting for the host to redraw the whole image.
The glasses have two 8 megapixel IR cameras working at different IR wavelengths. One is for tracking the gameboard and wand, and the second camera tracks what Tilt-5 calls “Tangible Objects.” Tangible objects could include game pieces and cards, some of which may have bar code identifiers on them.
Issues
It should be understood that Tilt-5 is a startup aiming at being a game system with a consumer price point, and the device has also had a long gestation period. I want to remind people that the Atari 2600 game system was introduced in 1977 for $189.96 or ~$877 in 2021 dollars.
The resolution is a modest 1280 by 720p resolution using a Citizen FineDevice field sequential color FLCOS with a custom projector with unique optical characteristics by Tilt-5. The effects of the field sequential color are evident but not that bad and sometimes show up worse in videos than they appear in real life. There is a lot of opportunities to keep improving the system.
There is some scintillation/grain from a beaded screen, and Ellsworth discussed how it could be improved with better (but perhaps more expensive) screen technology. Tilt-5 is not a technology designed for photographs or replacing a computer monitor, and the technology works best as a table-top display.
To some, the headset looks geeky and toy-like, but they want it to look like a game/toy system. I could see people putting “skins” on the headset or perhaps using retro-reflective tape for a dynamic skin.
If there is one thing I don’t like, it is wired to the host phone, tablet, or PC. I would like to see this wirelessly connected with a rechargeable headset. Cords are a snag hazard both for people and for damaging equipment and the cords themselves. Ellsworth says that the data rate to the headset is not excessive due to the processing in the headset, which would be possible in the future.
I am also not a fan of supporting the headset’s weight (about 95 grams) with just the nose and temples, and I would rather it had an adjustable headband. They could then put some weight, including a battery for wireless operation, in a pack on the back for better weight distribution (similar to the Hololens 2 – right).
A Little of the Physics
I want to briefly explain some of the physics behind Tilt-5 as I understand it below.
Left and Right Eye Stereo Polarization
The optics of the Tilt-5 is elegantly simple. The optics of the Tilt-5 is elegantly simple. With a combination of polarizers and quart waveplates, light for both eyes is selectively steered out to the retro-reflective-screen, and the only light to the specific eye is let through when it returns.
The LCOS projector outputs linearly polarized light that is rotated/retarded by a first-quarter waveplate on the projector’s output to a second waveplate that further rotates/retards the light so an angled linear polarizer will reflect the light. After reflecting off the polarizer, the light will be retarded again by waveplate 2 on its way to the retro-screen. The retro screen will not affect the polarization and will reflect directly back toward the glasses. The light is then retarded again by waveplate 2, which will cause the light to pass through the linear polarizer. The reason for the multiple passes through the waveplate is to set the light up to reflect the first time and then pass after reflecting off the screen.
The Tilt-5 will use different waveplates for the left and right eye so that the light is polarized in the opposite direction when it hits the screen. Each polarizer will only pass light for the given eye, thus supporting stereo vision.
The real world only has to make one pass through a single polarizer. A typical linear polarizer will block about 60% of the incoming unpolarized light and thus it will be about 40% transparent. This is better than most birdbath designs (such as Nreal and others) that block about 75% of the real-world light. Because of the waveplate arrangement, only about 25% of the projector’s light will be lost in the reflective and transmissive pass of the polarizer.
Something very different from almost every other AR headset is no “combiner” per se. Both the virtual light and the real-world light follow the same path to the eye. With a combiner, usually, the virtual path is reflected, and the real-world is passed/combined.
The Screen is Effectively a Very Large Optical Element
The last optical element must be as big as the FOV is at the distance it is from the eye. The size of the image projected on the screen sets the size of the FOV, and it can be as big as 110-degrees from the projector.
Retro Screen
On a normal white screen, light becomes highly and spread out over a 180-degree hemisphere, and only the tiniest percent of light will make it back to any one person. A retro-reflective screen sends most of the incoming light back in the direction of the light source over a very narrow-angle. A typical retro-screen is covered in tiny round glass balls glued to a gray background.
With the round glass beads (below left), light is refracted on entering and will hit the back at an angle such that about 20-30% of the total light will be reflected off the back from the bead. The light reflected off the back will be refracted by the side opposite to which it entered, and parallel to the direction it entered the bead. Since the projected image is coming from the direction of the eye, a high percentage of the light will go toward the eye. The microphotograph below is a picture glass bead from a retro-reflective fabric (similar in principle to what Tilt-5 is using). Importantly, the refraction and reflection process of beads will (largely) preserve the polarization of the light.
The three photos below show the effect of lighting the retro-reflective fabric from above, behind and from the direction of the camera.
Focus and Vergence-Accommodation Conflict (VAC)
I don’t know for sure, but I think there are two things at play with the Tilt-5 that help with VAC. The first is that the projector acts like a very high f-number light source, sort of like a pin-hole, and in turn, it may be easier for the eye to focus.
From the point of view of focusing, the light will appear to come from the screen. In Tilt-5’s normal table-top use, the screen is physically horizontal, and the user is looking down at it at an angle. Things appearing to be above and below the table/screen surface will still focus at the screen’s distance and not at their virtual position. But the difference between how far above or below they appear will be relative to the eye to screen distance will typically not be enough to cause a VAC problem.
Occlusion
Because of the retro-screen, the light projected from the Tilt-5 is comparatively dim and polarized. When encountering typical real-world surfaces like hands or other non-reflective objects, the projected light will be scattered and largely depolarized. Thus almost non of the projected light will be seen via the polarized lenses on the Tilt-5.
Shadows, or the lack thereof, is another curiosity of the Tilt-5. Normally with projectors, if you put your hand in the way, it will cast a shadow. But, because the projected light is coming from the eye’s direction, the same hand or another object blocking light that would cast a shadow keeps the user from seeing the shadow.
Multi-Player on the Same Board
Because of the retro-screen, each player only will see the light from their headset. Thus multiple players only see their virtual content individually while and real-world content such as playing cards and playing pieces are seen by everyone.
Courting Game Developers
Ellsworth talks a lot about working with many games developers and claims many writing games that take advantage of the Tilt-5. She has a background in working on consumer game products and knows the critical importance of having many people developing for a platform. The games end will sell the system. As far as I can see, the Tilt-5 is the only AR platform well-aimed at tabletop games, particularly with most other companies retreating to “enterprise” markets.
One Woman’s Vision – Should VCs Be Embarrassed?
Tilt-5 has a long history and is the story of tenacity and belief of Tilt-5’s CEO, Jeri Ellsworth. If you don’t know her story, I recommend listing the AR Show Podcast interview or searching for the many articles and videos about her journey. Ellsworth invented the retro-reflective AR concept at Valve and then took it out in 2013 as Cast-AR. When Cast-AR then folded, Ellsworth bought back the IP and founded Tilt-5. I can only admire how she kept going, and it is inspiring. Knowing the story and seeing the results, I can’t help rooting for Tilt-5.
I don’t know why it has been such a long struggle for Tilt-5 to raise money. But on the surface, the Valley VC and corporate funding entities, with all their talk of wanting to back women-led companies, should be embarrassed that they showered the likes of Magic Leap billions of dollars. At the same time, Tilt-5 had to go begging on Kickstarter.
Conclusion
On this blog, I “grade” products by how they fulfill their intended purpose. Tilt-5 is not trying to be all things to all people but rather a fun interactive game experience. It is doing a great overall job on that scoring, and the experience is magical and affordable for the intended audience.
Maybe I am jaded by many years of overhyped products, and I came in with low expectations for the Tilt-5. But I came away feeling like a kid with a great new toy which is how this type of product should make the user feel.
I’m also a bit frustrated that it seems that too much money has been poor on some overhype products while others get the crumbs. While I love Tilt-5 for what it is, I think it could be even better with more investment.
Karl, thank you for pairing a holistic picture of Tilt 5. I’m personally a huge fan and agree with not only your review but your commentary regarding investment. Jeri and her team are going to have a real impact and I’m excited for them!
Do you see potential for this technology in an office/productivity context?
Is the current – or currently possible – image quality (eventually) going to be competitive with font rendering on traditional displays?
E.g. for use of Microsoft Word or Visual Studio?
There are two tiers of “enterprise market – the high end / high cost tier of “display plus headset” that the industry is pursuing in search of margin, and the low cost “cubicle” scale market. There is also the aspect of CI for intermittent / low power / low weight portable markets currently held by cell/smart phone products,. In this context, stereo/3D capability might not even be required to make projection competitive?
I think the Tilt-5 concept has applicability beyond games, but not as, say, a traditional computer monitor. It works best in the “tabletop” type application. For instance, I can see it used in virtual “sand table modeling” (used by the military and elsewhere) where someone wants to visualize something like a 3-D terrain from above.
Tabletop use would pose a serious problem for other AR/MR headsets. First of all, most of them set the focus at 2 meters, and in tabletop games, the distances are going to be often less than 0.5 meters (at arm’s length). I find most of the Hololens and Magic Leap game demos as a bit contrived. It is problematic to develop a good “game” where you don’t know the shape and boundaries of the playfield. What can you assume about the shape of a room full of furniture, windows, pictures, etc.? They seem to mostly end up with things coming out of the walls you can shoot. Tilt-5 flips it around with what they call “AR Somewhere.”
I am not sure about Tilt-5’s “real world” content, like people and natural landscapes. I suspect they will look a bit sparkly due to the retro-screen. Tilt-5 has only demonstrated computer-generated images, so I don’t know for sure. Also, with 720p, the resolution is a bit low by today’s standards. So I would not want to be editing an MS-Word document using Tilt-5.
They did not seem to show many people at the Tilt Five booth, but I did get to see a volumetric video played on the Tilt Five unit at their booth as well as some of the computer generated graphics demos. They probably did not play it that much in their booth because Arcturus (who apparently made the video) was playing it in their cube which was right across from Tilt Five. It looked pretty good. I was able to get about 4 or 5 inches away from the guy in the video riding around in circles on a BMX bike. It started feeling pretty real at that point (except it looks like the video was maybe only encoded at 15 fps…). I’m sure you would pick up a lot of extra details about it if you try it at some point.
Thanks for the comments, Karl. I agree on “AR Somewhere” advantage/trade-off.
However, the weakness of other AR solutions for “tabletop” use cases does not really speak (IMO) to how Tilt-5 would perform for a “screen” or “wall” scenario (T5 calls this “vertical”, and removed support for it).
For example, I have seen reports that retro-reflected brightness varies with eye-board distance, an issue that is is certainly maximized with a board seen at a 30-60 degree angle vs. a “screen” seen almost perpendicular of optimum distance. That – pace Mark Bolas – can be software-corrected (reducing brightness for “close” slices) – BUT T5 “tabletop” is not by default superior to T5 “vertical” when it comes to visual fidelity or text legibility (if any)?
Do you think T5 itself will perform inherently better in tabletop, compared to itself performing on a titled or vertical surface?
I agee that the 720p rules this kind of application out for now. Could you cast some light – hah – on this related question: Is LCoS projector miniaturization played out for reasons of physics (cannot make pixels smaller)? If so, does that mean that a 1080p or 2160p Tilt-5 system would by necessity have to use projectors scaled by 3 in width/length and by 9 in area? That would certainly require a transition from “glasses” to “headseat” design?
Furthermore, I assume LCoS power consumption and waste heat also scales with resolution, which is yeat another trade-off that shifts serious use of this approach from “casual” to “involved” – less isolating than VR headsets or HoloLeaps, but not the same as, say, putting on some Active 3D TV glasses?
I also recognize sparkle and rainbow are concerns. OTOH I am guessing that there are opportunities for improvements to retro-reflective material properties and manufacturing that were not worth pursuing for established use cases.
Great article. I enjoyed getting a better understanding of how the optics on Tilt Five works.
Agree that Tilt-5 promises to be a best seller and also that Jeri Ellsworth’s journey will inspire many. However, not sure if they “had to go begging on Kickstarter” – many startups will make this approach a conscious choice, especially when the product is in the consumer space.
While your point is good, in this case, after the CastAR experience Ellsworth et al very much did not want to do another KS. Doing a KS for T5 was a last resort with the last of the money.
The reason for not wanting to do another KS at that point was that doing a successful kickstarter campaign with hardware like the castAR / TiltFive is because it is a very significant amount of time and effort. The good thing was that there was still a number of us that backed the castAR KS campaign and still backed this campaign and helped out the campaign as some of us are familiar with the underlying technology and still backed the idea and concept because we still believe it and the technology has a place in the AR/VR/XR ‘verse’ and would have a place and also look forward to the development of the next generation of these glasses in the not too distant future, where the resolution may be increased and other such features after this first version is launched and plan to develop software/games for this and future versions.
That s not strictly true. VC partners were interested but like many clueless investors didn’t see where the market would be. So a KS was a way to show customers are there. 3k+ are, and I know the pre-orders are flooding in too.
As soon as units ship (very soon) we will start seeing real world reviews and I suspect ‘magical’ is just one of the many superlative adjectives that will be used.
Did you get to try Lynx at AWE? Seems like a very interesting product for people who just want to prototype/”play with” AR.
Yes, I saw Lynx on Thursday. I plan on doing a write-up on it.
How would the experience compare with with older technologies that involved stereoscopic lcd table top surfaces?
I believe Microsoft had a prototype at one point.
Interesting question. Let’s consider an LCD stereo TV as a tabletop.
First, I don’t think 3-D stereo would work with an LCD tabletop. All the 3-D stereo technologies with LCDs require a person to be looking straight on and upright. This is true of passive or active 3-D glasses. Even most passive techniques, such as lenticular screens, have a limited viewable area. I think you would have to go beyond the capability of Looking Glass (https://lookingglassfactory.com/) to generate a true light field with a large enough view-cone/sweet-spot
Second, Tilt-5 gives each person sitting around the table a 3-D Stereo perspective based on their orientation. They are not limited to orienting everything the same way for all users so that things like labels and numbers/scores can be oriented based on the individual user.
Third, there are obvious size, weight, and portability issues.
Fourth, you would have to protect the heck out of the LCD. Remember the old Nintendo Wii and all the TVs that got broken (https://www.youtube.com/watch?v=D_uXEWKdsTY)? Likely you would want a Plexiglass or some other clear protector to keep from scratching and breaking the TV.
Next, the Tilt-5 has a lot of flexibility as to the size of the play surface.
BTW, Microsoft had a Tabletop “Surface,” rebranded as “PixelSense,” freeing up the “Surface” name for its tablets and touch surface laptops. The Surface started with DLP projectors under the table (see 2007 article: https://www.popularmechanics.com/technology/gadgets/a1690/4217348/) later moved on to using LCDs.
Examples: https://arstechnica.com/gadgets/2011/11/microsofts-table-sized-tablet-surfaces-in-pre-order/ and https://www.neowin.net/news/behold-the-5500-windows-10-coffee-table/. I don’t know of any of these supporting 3-D stereo.
I guess you could play board games on a large tablet computer, but I don’t see people doing it. I imagine people would not want the display getting all scratched up, plus it would be difficult for multiple people to participate.
I can’t guarantee that Tilt-5 will succeed, but it does not look like a tabletop LCD is a threat to their market.
So I understand most of the points, but there definitely was a large (auto)stereoscopic table format, in which each of the participants had optically tracked 3D glasses, and perhaps a wand.
It was several years back now, perhaps on display at Siggraph.
It was big and fixed, and I think the demo application was a skeleton, as viewed by a number of surgeons.
Curious what your thoughts would be how *that* would optically compare to this.
I understand the advantages this has to weight, portability, etc.
Wish I could find the reference.
I do remember it was constrained by the number of viewers obviously as the frame rate was divided by the number of participants.
I’m going to make a pretty confident guess that you’re talking about Euclideon, Rob. https://newatlas.com/hologram-tables-euclideon/50868/
I’m so delighted that you finally got hands on with T5, Karl, and that it hit with you. As a reader of just about all of KGOT and a fanboy and backer of T5 since the CastAR Kickstarter, I was aware of your (reasonable) dismissive thoughts on the tech.
Mostly this is just happiness about cool people finding out about cool things, but also I no longer have to write out the arguments for a comment I’ve been lacking time to edit for a couple years. Now I can just note the conclusion: Many of your articles over the years, and many comments by your readers, have pointed out something that you’ve felt was needed in display tech and particularly AR: An intentionally constrained device that is comfortable and actually useful for some small set of well defined use cases, at a consumer price point. If I’ve read your implications correctly, this product, almost regardless of what it actually does, could serve as an anchor for entire industries by standing in contrast to technophilic hype for its own sake. It’s seemed to me for years that CastAR/Tilt Five sat squarely in the middle of your specification.
Intresting, similar to VR pancake optics also using quarter waveplate and reflective linear polarizer.
Quarter Waveplates are a very useful element used in many different optics. Many if not most AR headsets have at least one waveplate(s). Most birdbath optics have them (Nreal, for example). Quarter waveplates let you switch/gate which light goes through or blocked, and liquid crystal lets you electronically switch the state.
Look at my teardown of Nreal (https://kguttag.com/2021/06/04/nreal-teardown-part-2-detailed-look-inside/). You will see they use them even though they start with an unpolarized display, an OLED, and you don’t need them for stereo vision. This is similar to their application with pancake optics, where they are using the waveplates to selective control whether the light will pass or reflect off of surfaces to roughly double the path length of the light.
Look at my recent Magic Leap 2 dimming analysis (https://kguttag.com/2021/10/28/magic-leap-2-pt-3-soft-edge-occlusion-a-solution-for-investors-and-not-users/) you will see they use quarter waveplates too.
How much a quart waveplate affects the polarization is retarded is also controlled by rotating the waveplate. This causes them to be used to improve the contrast of LCD devices to remove what is known as “residual birefringence.”
While the word “waveplate” makes it sound like some hard glass-like material, they are typically made by precision-controlled thin films of plastic.
One caveat, as far as wearable devices go, is introducing polarisation depenance automatically loses you 50% of your light, i.e. almost half the mass you’re carrying is wasted (bear in mind batteries tends to be quite significant in device’s mass budget) – of course using the F(LCOS) already has the polarisation hit. The other problem is incorporating polarising beamsplitters in wearer’s visual field yields the possibility that a number of display panels in the outside world can’t be read (LCD instrumentation, computer screens). Another issue is polarising optics are usually wavelength and angle dependant, introducing spectral shift at greater angles of incidence – however for this device its easier to correct than in pupil-splitting expansion schemes (ie waveguides) – unless using wire grid polarisers. On the otherhand the FLCOS is capable of very consistent contrast across wide Field of View and very fast swtiching, so its probably the best option given the lack of suitable emissive panels presently.
Overall good information. Using “real” rather than “ideal” components, polarization typically losses more than 60% of the light (about an additional 10% is simply lost in the polarizer).
Still often using polarization is MORE efficient than not if there are beam splitters involved due to the use of QWP. Birdbaths that start with OLEDs often use polarizing (see my Nreal teardown: https://kguttag.com/2021/06/04/nreal-teardown-part-2-detailed-look-inside/). With a 50/50 non-polarizing beam splitter you are looking at 25% after a transmission and a reflection. With a polarizing beam splitter, you lose about 60% on the first pass and 10% on the second pass or about 40% x 90% = 30%.
Euclidean sounds like a recent similar concept, but the multi-user autostereoscopic table I remember was likely the early 2000s.
Googled a few siggraph links, but could not place it. Thanks though.
A few facts from a fanboy that I hope KGOT readers (and writer) would enjoy knowing. One about the thing as a product, one list of illustrative videos, the rest technical and optical.
—
There are two “models”, LE and XE. In the KS and right now, the XE costs USD360, $60 more than the LE. It’s also currently impossible to preorder the LE, as T5 either never offered them for presale after the KS results or quickly sold out of their intended production run. (To the surprise of some people, slightly fewer than 1 in 8 KS backers chose the budget option.)
The glasses and wand are exactly the same across both “models” – they’re really more like trim levels. The two differences between the two levels are 1. a carrying case (at least according to the KS, it may have been dropped) and 2. a larger board, with some extra capability. The boards you see in the photographs here are LE boards, the basic configuration; T5 has always shown the system off with the basic board. It creates a space big enough for a typical board game and some room around it for virtual cards, mats etc. The XE board can do the same LE configuration, but the two rectangular halves aren’t connected to each other and the board includes a roughly equal “third half”. You can arrange the three pieces to make a larger board with a non-square aspect ratio. This is intended possibly for some bigger boardgames, but mostly for use as a battlemap for wargaming and tabletop roleplaying games. This configuration also allows, or possibly will allow after launch, setting the board up as a square with a propped-up wall behind it, to make the active volume taller above the table.
—
At one point I pulled together a list of the clearer CastAR/Tilt 5 through-lens videos in this Reddit comment: https://www.reddit.com/r/gaming/comments/isqv8i/tilt_five_augmented_reality_table_top_gaming_my/ge9ikjh/ I would note that in the last one I edited in, the Twitter video by Brielle Garcia, it turns out cardboard can be beta too. The first batch of beta backer boards tends to curl with changes in humidity. Apparently this has been addressed somehow.
Editing that list makes a robot moderator twitchy, so I’ve stopped. I would add two to the list here. Sorry for the call for copy-paste search, I don’t want to trip spam filters by linking too much.
Tweet 1403441015797600256 I just plain forgot when editing; it shows the arcade shooter Battle Planet by THREAKS running on a T5 board.
YouTube video 541QkQtuWxo is a recent demo, as in “demoscene”, implementing mostly classic Amiga-style effects for T5. Maybe skip to 1:30.
—
It’s so obvious to those who’ve understood the tech that it often doesn’t get mentioned: There’s almost no such thing as IPD or other setup with T5. You can don/doff the glasses in a second without worrying about alignment. In theory you’d still like the software to know your IPD because distances will be slightly wrong otherwise, but there’s no eyebox, no distortion if something’s off a couple mm. The worst-case image scenario for wide or narrow IPDs is that the image is slightly dimmer.
—
Because it’s working with a field-sequential display, the onboard reprojection runs not just on every frame but on every *field*. Typical objects and scenes with a reasonable mix of R, G, and B are effectively stabilized at 180Hz, twisting a constraint of needing to use an FSD into an opportunity for better performance. This also cuts the field-sequential rainbow edge effect by a factor of three for head turning relative to the scene, though not for fast-moving objects within scenes.
—
“Because of the waveplate arrangement, only about 25% of the projector’s light will be lost in the reflective and transmissive pass of the polarizer.”
Close. Ellsworth has claimed repeatedly that 85% of the light coming back from the board gets through the polarizer. I include that super minor difference only in case it implies something interesting about the tech.
—
At some point a “polarization sensitive reflector” – from my skimming, some kind of Bragg grating thing – has been incorporated into the lens, allowing T5 to remove a polarizer from inside the light engine, where it’s expensive to insert and delicate to align. Apparently this kind of coating or film has only become economical in the last few years.
—
The process KG talks about with the waveplates changing the polarization is often summed up in the jargon “circular polarization”. The two projectors and lenses are oppositely circularly polarized; it’s fundamentally the same system used by 3D theater projection. The big reason for converting to and from circularly polarized light is to avoid the interference and oddities that linear polarizers can cause with things like LCD screens – and that are much worse when you have them only in one eye.
—
A feature that’s been dropped for launch next spring but is very much intended for implementation down the line is timesharing – the ability for one executable on one device to divide its framerate between multiple headsets on a USB3 hub. In situations which can tolerate a fraction of a second of lag in response to input – everything but arcade games – this leans on the reprojection feature to support multiple headsets within pretty much the compute and bandwidth demands of a single set.
—
T5 has fairly recently made it explicit that they point the two projectors a bit outward, because pointing them both directly forward would waste render and bandwidth projecting pixels in a crosseye zone that people can’t see comfortably and many can’t see at all because noses exist. This has implications for the FOV (wider and shorter than the naive case) and PPD (sharper).
These numbers are speculative. Halfway through CastAR, they started quoting a 110 degree diagonal FOV. They also on a couple of occasions stated that they were running their 720p projectors at “a bit more than 70 degrees horizontal” FOV. The tech is “easy” to adjust to a desired FOV, so it seems likely the team could decide early on their compromise between FOV and resolution. T5 is still using the same diagonal FOV number. For these two reasons I think it likely that the FOVh number still applies. I also spent a week finding out that all the complicated tan-atan trig calculations for FOV come in with flat screens; given that you’re working purely with angles it apparently is just as simple as the Pythagorean Theorem and multiplying by aspect ratios? If so:
Numbers for naive case, full overlap pointing forward:
Horizontal FOV: 95.9 degrees
Vertical FOV: 53.9 degrees
Pixels Per Degree: 13.3
Pixel side length at my fingertip reach of 70cm: 0.92mm
Numbers for estimated true case:
Single projector horizontal FOV (postulated): 72.0 degrees
Single projector diagonal FOV: 82.6 degrees
System horizontal FOV: 102.2 degrees
Projector and system vertical FOV: 40.5 degrees
Projector outward pointing: 15.1 degrees
PPD: 17.8
70cm pixel side: 0.69mm
Ugh, no, those numbers are wrong. Of course I only thought of the analytical sanity check after posting. In spherically projected geometry an equilateral right triangle exists, so the PT obviously cannot apply. FWIW, I only abandoned the tangents route because I did horizontal and vertical FOVs independently as a crosscheck and couldn’t get square pixels, an assumption built into the problem. I’ll go learn more.
Thanks, you have a lot of interesting information with many good points—also lots of the things I will also want to test out many things when I get one.
A few things:
– I’m particularly curious how well they can correct for field sequential color with head motion, and I know others have proven it can help.
– I think the difference in light loss is that there is about a 10% to 15% light loss on the bounce out and another 10% to 15% light loss on the way back through. I’m not much concerned about the exact numbers.
– I don’t understand your point about “a polarization-sensitive reflector.” This sounds like a normal plate polarization beam splitter (just with fancier language) of which there are many different types with different efficiencies and costs. The LCOS projector has to have a polarizing beam splitter in it for it to work at all. The light illuminating LCOS has to be polarized, and the beam splitter polarizes the light on the way in and “analyses” it on the way back. There may be additional “clean up” polarizers. Either way, the light out of the projector is highly polarized, and thus there is much less loss with the larger lens circular polarizers.
– A “circular polarizer” is simply a combination of a linear polarizer and a quarter waveplate. As you wrote, it does significantly reduce, but not eliminate, the effect of a highly linearly polarized display (ala, LCDs).
– Thanks for all the calculations and “showing your homework.” I will be curious to see the actual field of view of the projectors. There may be some marketing fluff in them. I’m also interested if the retro-screen has an effect on resolution.
“– I’m particularly curious how well they can correct for field sequential color with head motion, and I know others have proven it can help.”
It should be more than possible, if I recall there are border pixels on the FLCOS (if they’re using the panel I think it is) for shuffling the image about (sequentially), and its very fast swiching. In that scenario the limitation is the head tracking sensing and algorithms (ie kalman filtering feed forward)
When T5 first described the reprojection feature, I also thought in terms of shifting the image to the nearest pixel. In fact they’re doing something much more sophisticated and, from a certain point of view, straightforward. At every field, their chip looks at the contents of the framebuffer and reprojects it in six degrees of freedom onto the pixels of the projector based on where their sensor fusion says the glasses are.
Image shifting can only deal with pitch and yaw; this approach handles pitch, yaw, and roll, and small translations along any axis. It takes more chip, but the same or less development effort. Here’s Jeri Ellsworth describing how the feature works and demonstrating at 0 fps the feature’s robustness under rotation and its failure mode under large translation.
https://www.youtube.com/watch?v=0zfxrnArr1w
Ellsworth refers to the “Archie” chip in that video – it’s the predecessor (or now partner?) to the Movidius. Her history is in price-sensitive toys and consumer electronics, and her specialty is building performant systems on commodity gate arrays. Her big start in tech was designing a commercially successful hardware emulator for the Commodore C64, which apparently is especially difficult to emulate. In between CastAR and T5 she made the flight computer for Astra, the containerized smallsat launch company that just reached orbit. Archie was the name for the SoC she had doing just about every calculation on the glasses, including the reprojection. It’ll be interesting to learn someday why they shifted to the Movidius; my guess is it’s to offer developers the possibility of hardware acceleration on object and hand recognition.
One little oddity about the system – T5 doesn’t rely on stabilization nearly as heavily as VR sets or AR sets with distortion/FOV/eyebox problems; it’s marketed as a board game display and it’s AR, it’s not going to make anyone sick if it jiggles sometimes. Nonetheless, this low-spec HMD is possibly the first ever to put CPU-independent hardware motion compensation out at the framebuffer, where it belongs. It seems unlikely, but I haven’t been able to find a counterexample. Quest 2 might have dedicated hardware, but their marketing all talks about their same CPU-based “Timewarp” and “Spacewarp”. I know the Mali D77 can do this for eight planes at once, but I don’t know if anything’s been released actually using the D77. Presumably other hardware uses the Movidius; someone else could be using it to reproject, though Intel’s marketing is all based on smart cameras.
“Because of the waveplate arrangement, only about 25% of the projector’s light will be lost in the reflective and transmissive pass of the polarizer.”
Worth pointing out that 50% is still lost initially at the FLCOS. One can try “polarisation recycling” of the incident illumination but this leads to much larger optics – to cover those field angles the beam will be expanding rapidly – you need to somehow capture it all, rotate polarisation by pi/2 and refocus back through the FLCOS.
Yes, that is why I said “the projector’s light” which I assumed was polarized coming from the FLCOS projector. Also, there is typically about a 10% loss of even light that is polarized correctly. So a polarizer of unpolarized light only lets through about 40% of the light. If light is “correctly polarized” then typically about 90% will get through with a 10% loss.
On large projectors, it made sense to use polarization recovery/recycling, but even then is gave only a slight advantage. I don’t know of small projectors used in AR using polarization recovery. So you lose about 60% (50% for polarization and about 10% is lost) for polarizing. The first time a person digs into how displays work, they can be shocked by how little of the original light gets through. For example, typical LCD TV only lets through about 5% to 8% of the light from the backlight.
[…] my last article, I discussed my thought on Tilt-5, which I saw for the first time at AWE. For this article, I’m grouping ST Microelectronics […]
Correction: It’s “Norman Chan”, not Chang.
Also, the VCs need even *more* admonishment because if I recall correctly, they originally invested in CastAR, after a successful Kickstarter and well into development, and pulled the plug on them.
Thanks. Corrected Norman’s name.
Listening to Jeri, there were several problems include some bad advice on who should run CastAR.
[…] AWE 2021 Part 1 – Tilt-5 Was Magical – KGOnTech […]
[…] AWE 2021 Part 1 – Tilt-5 Was Magical – KGOnTech […]
Not according to Ellsworth. Her stated reason is that it’s awful and restrictive to owe something to thousands of people, and to have thousands of amateurs complaining about every obstacle and questioning every decision.
https://youtu.be/Bn5qhJv3L0Q?t=5328
“I did NOT want a new Kickstarter the second time. Like the entire time we were bootstrapping Tilt Five, I was just saying like, ‘We are NEVER doing Kickstarter again. We are NEVER doing that.’ There’s nothing that feels worse than having… three or four thousand people, all with like guns pointing at your head, like, ‘Deliver the product, tell us why it’s late’…You know, it feels terrible being part of a Kickstarter because most- a lot of people that join Kickstarter don’t understand that it’s like you’re trying to look into a crystal ball and figure out…”
I don’t know why this replied to the wrong thread. I definitely didn’t click on a different button several pages off.
Hi, Karl
samrt solusion. But does this technology solve any real problem? I means basically it’s a 3D display with 2 projectors. any dlp projector with shutter glasses can do it on any surface, am I right? or I miss somethings.
I think that Tilt-5 is primarily for tabletop gaming and what is known as “sand table” displays (where, say, you model something on a tabletop – see for example https://en.wikipedia.org/wiki/Sand_table). It is definitely not general-purpose AR.
It is wonderfully simple yet effective for what it does. Because the “focus point” of the light is at the screen distance, you don’t have VAC problems. The 3-D effect seems to work quite well. Because the light is fairly dim from the projectors, it is only visible where you have line-of-sight to the retro-screen, so occlusion of the virtual by the real world works almost perfectly (but there is no way to occlude the real world by the virtual). Everything lines up well because the projection comes from very close to your eye (via a beam splitter). For being such a simple concept, it works well. It will not deliver perfect image quality as the screen is beaded. It does scintillate (sparkle) a bit.
I had not seen their demo before AWE 2021, and I had low expectations. I was surprised by how well it works.
Karl
Hi Karl,
I recently spoke with one T5 independent developer and they mentioned that while T5’s retro projection surface is indeed a magical thing to behold, it isn’t perfect. I think you’re hinting at this with your observations of scintillation. This is probably due to the quality of the manufacture of that particular screen. Perhaps the glass beads were not packed closely or uniformly enough such that there are gaps in the material that can cause light scattering. Once the industry starts to see the value of retro projection technology, hopefully we’ll start to see rapid investment into higher-quality projection surfaces.
I have thought about the issue of bead size and uniformity as well with the Tilt-5. Jeri Elsworth, Tilt-5 CEO, told me that certain coatings would also improve the image quality.
While there are probably ways to reduce the effect, I think that some amount of scintillation may be inherent in the retro-reflective screen.
Hello Karl,
You wrote: “It (to some degree) addresses vergence-accommodation conflict (VAC)”
This is different from Tilt-5’s position – I believe Ellsworth herself stated that the Fergason 1997 approach they are using now essentially “solved the Lightfield problem” and removes any conflict because vergence and accommodation.
I am having a hard time finding my own answer. On the one hand, I can argue that (in the ideal case) the mirror places the virtual image of the projectors at the user’s eyeballs, and some large fraction of the emitted light (reduced by divergence in the beads) will converge back onto that focal area perfectly – in other words, the user does not focus on the beads, but on the object “behind” or “in front” of them. Their video
https://www.youtube.com/watch?v=RnOLuhBOU8s
certainly claims this.
Whereas you appear to say that the lack of VAC is due to the lack of “board space verticality” of the scene rendered? For tabletop, verticality down is certainly clipped by the closer edge of the “bowl”, and verticality up is clipped by the upper plane of the view frustum through the far edge of the board. But assume a board on the floor seen perpendicular by a user looking straight down – at 2m distance, a virtual object placed 1m “off the user” (or 1m “behind the board)” would, according to you, produce VAC strain, but according to Ellsworth would not be a problem at all?
If they are right, this optics would be fundamentally different from e.g. 3DTV, as it would inherently product correct depth of field.
If you have a reference that would help me understand the pin-hole/aperture argument you made, I would much appreciate!
There are a number of things about T5 that are very simple to say almost correctly, but much more complicated to fully explain, and in which the simple summary is more positive to the product. T5 is an AR device, an industry that generates huge volumes of marketing that is usually outright lies. Combine the two, and yeah, T5 (for which Ellsworth does most of the talking) sometimes stretches the truth. It’s an order of magnitude less than any other AR company, but still.
In short, T5 can show you an object far behind or in front of the material. It can show you an object without VAC if the object is near the material drawing it, which is where the eye accommodates. It cannot do both for the same object at the same moment. As you correctly understand from Karl, an object 2m away in 1m distant material would produce VAC. As you correctly understand from Ellsworth, in its primary marketed use case of tabletop display of virtual boardgames and videogames that happen more or less on a plane, that scenario has no good reason to occur. Thus, within its chosen applications, T5 produces practically the same result as a light field display without the hardware or compute costs.
[…] by far larger than any other optical AR headset,” notes AR displays expert Karl Guttag in his write-up about Tilt Five. “What makes it ‘magical’ is that everything seems to work the way it should in a […]
[…] Tilt-5 (briefly – see my article on Tilt-5 from AWE 2021 and SadlyItsBradleys videos on Tilt-5 for […]
[…] Tilt-5 (briefly – see my article on Tilt-5 from AWE 2021 and SadlyItsBradleys videos on Tilt-5 for […]
[…] Tilt-5 (briefly – see my article on Tilt-5 from AWE 2021 and SadlyItsBradleys videos on Tilt-5 for […]
[…] Tilt-5 (briefly – see my article on Tilt-5 from AWE 2021 and SadlyItsBradleys videos on Tilt-5 for […]
Hi Karl, Are you still using the Tilt 5? Has your opinion changed since receiving and playing with the unit more? I would be interested in a follow-up article if you have enough material for that.
I still have the Tilt-5, I got distracted with some other activities and keep meaning to get back to it. I’m interested in comparing it against say the Quest 3.
Note, I’m more interested in the display and human interaction aspects (its “SLAM-like” attributes) rather than the game play.
A comparison with the Meta Quest 3 would be awesome.
The current game line-up does not interest me much. To me a killer app would be remote board game play in which the physical pieces on one board would be mirrored on another remote board, and vice versa. I would not care so much about text readability, graphics fidelity or anything else.