Chapter 1. How AR Is Helping Manufacturing (and Other Uses That Show Promise)

When I wrote my first book, I focused specifically on augmented reality (AR) for industrial use. I did so because it has been the area, thus far, that has shown both the most promise and the most practical application. In manufacturing and industry, AR creators have piloted and rolled out applications that are saving people time and money.

Now we are at a new moment. Pokémon GO helped people—broadly—understand the idea of AR (even if it was only “sort of” AR). Facebook and Snapchat are now beginning to do that, too. The push toward AR by iPhone and iPad will create a similar tipping point among mass-market consumers. It’s not about the new AR-enabled iPhone itself or the splash it did or didn’t make upon launch—it’s more the wave it launched that is building momentum as ARKit spreads and more companies follow suit. Apple has a developer base of more than 275,000 people. That is clearly leading to the development of more and more AR apps. But it’s not the hardware, software, or even those apps that are making this an important moment in the development of AR. It’s that Apple championed the space so vocally that it motivated other big players to move into AR—as a strategic move. The big-company AR land rush has already begun.

On August 29, 2017, Google debuted ARCore, which combines motion tracking and contextual understanding of the environment to make it easy for Android developers to play in AR (which could ultimately be huge when paired with Google’s visual object search libraries to pull in content). Google also worked with Lenovo to create the Phab2 Pro—a smartphone with native AR features that cements its move in the space. As you’ll read in the interview with Vitaly Golomb later in this book, HP has made AR a strategic priority—indicating that it anticipates desktops will go away or entirely change form.

Microsoft integrated the team formerly at Altspace VR, who are experts in bringing people together in virtual environments, creating large-scale social virtual reality (VR) events. In early October 2017 at an event in San Francisco, the company announced its intention to erode the barrier between the digital and physical worlds by making a full-court press in its vision of mixed reality, and announcing a partnership with Samsung. In addition, Microsoft has moved from just the HoloLens and VR headsets to new types of controllers for AR and VR—a rudimentary, geeky, and necessary first step into the very interesting area of device-free gestural control. The first version always looks clunky. Just look at where computers started as illustrated in Figure 1-1.

All of which is to say that AR has become valuable territory for the big players to own.

But owning it and using it to help improve lives are two different things. And talented AR creators can aim high enough to do both. In this book, I look at use cases, within the world of commerce and outside of it, and bring you examples of how.

What Makes AR Compelling Now?

AR is not new, but its evolution is entering a new phase. It has become part of what heavy AR and VR investment company Rothenberg Ventures calls “frontier tech.” It is being billed as one tool in the larger bucket of mixed reality, blended reality, or—ultimately—just technology for creating, enhancing, and building human understanding of reality itself.

In terms of its development stage, AR is still a caveman club compared to the holodeck promise of what it will become. However, that is the 30,000-foot eagle-eye view, looking from the future backward. Looking at where we are now, the technology is actually in a great place. We’ve moved past the initial-hype-then-disappointment-when-challenges-hit phase of any new technology. We’ve made it to the end of the phase that usually follows that: one marked by rapid growth and then significant waiting. We’ve now made it to a point of steady growth—an upswing that has been driven by two things.

First, the hardware that drives the technology has made developmental leaps, particularly in the past two years. Processors have become faster; computer imaging has advanced; multiarray cameras are making the depth-sensing necessary dimensional experiences easier; and the camera technology itself is improving. Second, thoughtful AR makers have been steadily testing and refining how to use the technology, and they’ve become even better at it. The core of garage-makers and corporate AR leaders who have invested 15 to 20 years in AR, quietly building, testing, and refining, have brought AR technology to a clear and stable place. These pioneers deserve applause for their tenacity in doing that. And now, building on and refining their work, turnkey tools like ARCore and ARKit have made development much easier. Together, all of these advances in hardware, software, and thinking have converged to create momentum, pilots, and real proof of viability of AR. We are now beginning to prove that the technology has business applicability—and, as important, the ability to help humans.

And more mainstream companies—paddling out to get ahead of that wave—are getting on board. Throughout 2017, Facebook quietly hired an entire team of developers working on AR (among many other things). From its emerging-tech-focused Building 8, the company is slowly trickling out fun interactive tools, and in April 2017, its AR Studio also opened its Camera Effects Platform for developers to play along.

Snap, Inc., which is an increasingly notable player in the AR space, acquired Cimagine in December of 2106. What this means is that it now has some serious computer vision talent on staff, along with a technology that allows consumers to visualize products in new ways (which might translate to something more than silly filters—it feels like a new play entirely).

In June 2017, Apple acquired SensoMotoric, a German computer vision company focused on eye-tracking. It also purchased RealFace, an Israeli cybersecurity and facial-recognition company; Apple’s competitors have purchased similar facial-recognition companies, as well, which will help make their AR creation capabilities even more sophisticated. If you have basic web development skills, you can play with that yourself in some fun ways using open source files and playing with the face (camera) code at Trackingjs.com. Matthew Hallberg has a great video tutorial showing you free ways to do AR facial recognition.

A lot of these are “fun” uses of AR, but they’re not the only ones.

AR is broadening in awareness, acceptance, experimentation, and actual usage—beyond the world of manufacturing and industry. There are many potential commercial, educational, and informational uses of this technology—as demonstrated by the list of use cases. We’ll look at a few of those later.

First, let’s make a distinction in terms.

It Is Important to Be Clear: AR and VR Are Not the Same

While some people are now referring to AR under the umbrella terms “mixed reality” and “blended reality,” the language of this category is still evolving. One thing you should be aware of is that much of the discussion still lumps together AR and VR. When you see that happening, you should pause and mentally parse them. Because the fact is, the two technologies are distinct.

This blurring together of categories originally happened for a lot of reasons. Some AR systems use the same eyewear and headset tech as VR. Early industry analysts wanted to make the industry look bigger (or didn’t yet know how to parse it). Some early makers of AR are people who came from the VR world—and that is still often the case. The blending of these two terms understandable.

However, it is time for a clearer separation, both because of form factor (AR is not dependent on head-mounted anything) and because of their essential functions. Whereas VR brings you into the digital world, AR brings digital information into your world and overlays it onto the physical world around you. While the two technologies CAN play together, they don’t have to. They do different things in different ways.

VR is a closed environment that limits your view of the world—literally. AR puts digital content into your world, opening up your field of vision as well as more possibilities. And AR can be implemented using much simpler, existing technology—phones and tablets as well as new technologies. That matters when considering testing new tech. Using existing hardware makes sense for people seeking a quick or low-cost solution.

Both technologies will lay the groundwork for what eventually becomes something entirely different. But for now, while we’re still talking about head-mounted movies versus view-through-the-phone holograms, the two technologies are different enough to warrant individual consideration.

Why Should You Focus on AR Now?

AR represents a new way of seeing—and seeing into—the world. In this context, it is a powerful tool that “augments” not just the world around us, but more importantly, in a business context, adds to the abilities of humans and sheds greater insight into machines and products.

For instance, for the past few years Mercedes Benz has been putting QR codes in the B-pillars and inside the fuel door of all its new cars. The reason: so first responders at accidents can connect with an AR app to see color-coded diagrams of wiring and fuel systems. As they cut through these systems at accident sites they can do it quickly and accurately—saving lives. That is the power of AR, used thoughtfully.

There are now dentists using AR apps to 3D scan people’s mouths and virtually preview dental implants and enhancements. Room measurement tools like MeasureKit and TapMeasure are creating apps that make home renovations simpler—allowing easy measurement of interior spaces and 3D room modeling.

Later in this book, we look at other applications for which AR has been used in meaningful ways—from shipbuilding, to medicine, to art. The business applications of AR are myriad.

The question: why is now the right time to invest in this (hasn’t AR been around for a long time)? The answer: because now not only have the base technologies underlying AR changed, but so has the world.

AR Has Reached the Usability Phase

AR can help explain and visualize things in a way that is most natural for our brains to process, and do it more clearly than written instructions. For instance, to show someone how to operate or repair a machine, you first need to be able to visually and accurately show the machine. We can now do that (more or less easily), drawing from the digital world. More important, we now also know how to translate these digital images into tools that can overlay and display the important bits of detail in front of us. We can “holographically” display and render objects accurately. And we can have exactly—and only—the bits we want, exactly when we want them.

As Arthur C. Clarke said, “Any sufficiently advanced technology is indistinguishable from magic.”

Workforce Transformation: AR for Knowledge Transfer

AR is a tool that can smooth a socioeconomic issue that faces many coutries around the world: the aging workforce and shortage of skilled manufacturing labor. It’s an important area to consider. The latest data shared publically on the state of global manufacturing, a joint study by Deloitte and The World Economic Forum, forecasts that the manufacturing industry can expect a shortage of 10 million workers globally. Highly skilled workers are growing older. Baby boomers are retiring. Teachers, workers, artists, and knowledge-holders across all industries are leaving. And as they go, so does their knowledge. Companies need to either retrain for those skills or connect in with that brain-trust in new, flexible ways.

AR can help with that.

Matt Sheridan is a senior director who works on the ThingWorx platform at PTC, one of the largest industrial AR companies around. When I interviewed him, he described the situation this way:

What you’re having is these Baby Boomers are retiring. And the students that are coming out of school today don’t have the same skills that a 30-year veteran has. Why is that relevant for augmented reality? Augmented reality provides an ability to transfer that knowledge. You’re able to take the information that’s in the heads of these highly skilled workers and digitally capture that before you lose that talent. That can be in the form of service—[for instance] augmented work instructions on how to do a complex repair of a piece of equipment, it can be [used for] training and simulation of how to even do jobs before they go into the workforce.

The other thing to point out is that the next generation entering the industrial workforce is used to digital interfaces; they grew up on video games and with digital devices. Being able to give them a toolset they’re comfortable with to do their jobs is important, and AR really plays to that very well. A lot of the user interfaces (UIs) in wearables were taken right out of the gaming world. You have such things as optical gaze—it’s very similar to a Wii remote control, allowing you to move around and use your body. It also allows you to recognize objects and engage with them. This next-generation UI lends itself well to this new workforce.

Sheridan describes the positive impact AR can have by citing a study done by Boeing and Iowa state to reinforce the point. It’s a study I’ve written about before; however, Sheridan explains it very well:

There’s a study that Boeing and Iowa State did together. It’s a great study. It’s a benchmark for the way people talk about the benefits of augmented reality. Essentially what they did is [take] two groups of 50 students and they said “We’re going to do a study. We’re going to look at how Boeing builds a wing today on their operational floor.” Basically, the way Boeing [did] it is they had printed work instructions in a room outside of the manufacturing floor where you build the wing. You go into the room, you get your information, you go back out onto the manufacturing floor and you carry out your task.

They had these students who’d never built a wing before [build it from scratch]—it was a simulated wing, a wooden mockup—go in and do that.

Then they said we’re going to repeat those instructions with a [desktop with instructions] and with a simple iPad we’re going to digitally overlay the content [using AR] onto the actual wing assembly. We’re going to show you step-by-step how to put it together.

There were a lot of statistics that came out of that. The two that I thought were very interesting were: [having] augmented work instructions, they were able to do their job 30% faster. Augmented reality reduced the time it took to actually build the wing. But the really amazing one is: the first time they did it using augmented work instructions, they reduced their errors by essentially 95%. Here were these students who had never built a wing before on an aircraft, who were in school—they were able to do their job with 95% fewer errors than the team that actually was following the work instructions that Boeing has out on the manufacturing floor today. And then they did the study a second time. [In that trial] the students who built with the hardcopy work instructions had the same amount of errors. The team that used augmented reality had zero errors. They essentially eliminated all the errors from their work stream. That really was a game-changing statistic. To say, “Look, with this knowledge transfer we can impart to this new workforce, we can eliminate errors through giving them digital work instructions in context with the actual physical device.”

As Sheridan and I also discussed, SRI conducted a similar study that shows you can significantly reduce errors and work much more quickly.

Handheld and head-mounted AR tools provide the ability to transfer knowledge—even to unskilled workers. They allow you to collect and preserve information that lives in the heads of employees who have been around for decades and digitally capture it in a few ways:

Augmented instructions: By gathering information from a skilled worker, you can develop augmented work instructions about how to do a complex repair on a piece of equipment. You can easily use that in training and simulation, teaching people to do jobs before they go into the workforce.
Remote guides: AR can also allow skilled workers to act as remote guides for less-skilled workers. This creates the possibility for “retired” workers to reenter the workforce, part-time from anywhere. It also makes it possible for one highly skilled employee to have “many hands,” working through less-skilled workers (using AR systems and visuals as guides) to multiply the highly skilled employee’s value. It is now possible to be in many places at one time.

But that’s not the only way that we can use AR, and there are many interesting and positive use cases emerging! We can use AR in agriculture to visually map crop conditions to increase yield—and better manage production of food (or other types of plants.) Early prototypes are already being rolled out that make this happen. In the case studies presented in Chapter 2, we explore how AR can be used in art to make museum experiences more interactive and interesting in galleries and public spaces. It can make paintings come to “life” in new ways. And businesses from construction to automotive can use AR. As this concept car BMW debuted in 2016 shows, one day, it will make a pretty nifty windshield addition. (It’s already happening in small scale with products like the newly-revived Scully motorcycle helmet, relaunched in a wonderful way at CES 2018.)

Billion-Dollar Companies Are Making AR an Investment Priority

Among the historical mega corporations making major investments is HP. Vitaly Golomb leads global investments for HP Tech Ventures, the corporate venture arm of HP, “Silicon Valley’s original startup.” He launched his book, Accelerated Startup, in March 2017. He is a Global Corporate Venturing 2017 Rising Star, a contributing writer for TechCrunch, and a mentor for a number of startup accelerators.

Why is a company like HP (a billion-dollar company) making AR an investment priority? Here’s what he said when we talked about it:

As a venture group, we invest in preprinting, that’s an existing business that we support. [For this] the immersive categories are becoming increasingly important; this is mixed reality, VR, AR, everything related to that: component technology, software, content.

We’re seeing related initiatives for mixed reality popping up in different kinds of units. We have new display technologies within healthcare. We have, certainly, the consumer PC world with gaming becoming an important factor. Obviously, VR is leading there. We also have certain commercial groups that are interested in various applications.

HP is number one in PC as well as consumer and commercial—it’s a major business. It’s the bigger half of our business right now; it’s obviously important to not be disrupted because computers are going to change the way they are, substantially, in the next five to ten years or less.

The venture group…we’re the tip of the spear. I’m in the office maybe one day a week and I’m out there talking to companies. I’m out there constantly globally. I go to 20, 30 countries a year, so I see a lot of new things and, by definition, if it’s seed stage it’s probably about two years before any large company would see that technology in the competitive landscape.

Beyond just strategic advantage, why does AR itself matter so much to a company like this? According to Golomb’s thinking (and HP’s strategy), computing companies need to view this as a serious threat.

Golomb puts it this way:

Augmented reality is a much, much bigger deal [than VR] because it will replace all of our computing paradigms and all of our display paradigms—our phones, our laptops, desktops, and TVs on the wall. All those things are going to go away and they will be replaced with augmented reality—some form of seeing through something—which, most obviously, is some kind of glasses. Then, later on, contact lenses that will put things in our field of view. There are a few companies, like Samsung and Google, working on that. Ultimately, it’s something that you tap into the nervous system and display it without going through the occipital lobe, going through the eye—kind of tricking yourself into becoming a hybrid. This is obviously sometime down the line, but definitely within our lifetime.

It’s an increasingly important area for us and it’s more than just a buzzword, it’s more than just entertainment; it will affect all computing form factors seriously as well as retail.

Now that we have a sense of how AR is developing, the next big question to ask is, “How should I be investing in it?” Whether you’re a CTO, a CEO, a developer, or curious about the space, you need to know where to invest your time, attention, and money.

In Chapter 2, we look at just that.

Get Changing Realities for Business now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial