Creating Augmented and Virtual Realities

Part III. Hardware, SLAM, Tracking

The augmented reality (AR) user experience is compelling and often feels like magic. At its best, though, you shouldn’t even notice it at all.  The better AR system designers do their job, the less you notice their work, and you can focus on the content and interactions that help you to achieve what you wanted to do in AR in the first place. Solving the technical problems to achieve this is a very difficult problem; huge progress has been made, but many problems remain to be solved. This section aims to help explain how everything under the hood works, how we got here, and how to make choices as to where to invest your energy going forward. Hopefully, this chapter helps to clarify why, when the system seems to break on you, what’s going on, and give you some clues on how to design around that. For the next few years, building AR apps is going to heavily depend on how AR developers build products that work within the constraints of the systems while the system builders work to eliminate those constraints.

We cover the core technology underpinning all AR systems, simultaneous localization and mapping (SLAM), and why that’s a broad term that doesn’t really help explain anything! We address the components that go into a SLAM system and the limitations of these, plus we look at how some of these limitations (e.g., SLAM maps that are bigger than one device can handle) are being solved via the AR cloud to enable experiences like shared content, persistent ...