Augmented Reality for the Industrial Enterprise

Chapter 1. Why AR and Why Now?

Augmented reality (AR) is not new, but its evolution is entering a new phase—which I think of as “finally usable by a broad group of people.”

We’ve passed the “trough of disillusionment” that comes with any new technology. Big-name companies are investing heavily in the space, as are individual investors. The hardware that drives the technology has made leaps, particularly in the past year and a half. And most importantly, thoughtful AR creators have been steadily testing and refining how to use the tech—particularly over the past 15 years. We are finally past the pilot phase. And we are now beginning to prove that the technology has business applicability—and, as importantly, the ability to really help humans.

Why should you start planning for this now? Because it can help your workers. And it can save you significant amounts of money.

Companies who are ahead of the curve are already cementing their leadership in the AR space—and creating using the tech. (Intel purchased Recon last year as a way to get into the game. PTC purchased Vuforia from QualComm. And in May, Apple snapped up German AR company Metaio.) Those who are currently following the work of Magic Leap (acquired by Google for $542 million) can expect a slow progression toward what will ultimately be a huge leap forward in technology.

All companies should be looking to augmented reality as a key tool for creating the future. And they should start looking now.

Why?

Someone is sitting in the shade today because someone planted a tree a long time ago.

Warren Buffett

The best way to predict your future is to create it.

Abraham Lincoln

Pokémon GO Was a Fad—And Also a Tipping Point

Industrial enterprise AR is not the same as consumer AR, but the two are interwoven in a way that pushes both of them forward. The more people—workers, kids, college students, ordinary folks—who understand and embrace a new technology, the faster it is embedded into collective consciousness. And, in turn, the easier adoption becomes. A rising tide floats all boats; in this case, the tide looks a lot like a Pokémon.

On July 10, 2016, Fortune reported that Pokémon GO was about to surpass Twitter in active daily users, and that the game had been installed on 5.6% of all Android devices in the United States. After that, those numbers spiked. There have been 10 million Android downloads. The app generated an estimated $1.6 million a day from the iOS App Store. (Each day. Just on iOS. Just in the United States.) Nintendo’s market value increased an estimated $11 billion in the first five days after Pokémon GO was launched. That number went up a lot, then dropped a lot, but after all was said and done, it still made a whole lot of money (and was a massive step forward for AR).

Is and was Pokémon GO a fun, nostalgic fad? Yes, absolutely. Will some businesses look at the Pokémon model and make strategic mistakes, like trying to make people chase floating cars as AR advertising? Yes, they will. And that doesn’t matter. What Pokémon did is embed—finally and clearly—in public consciousness what AR is and how it works.

Why lead with this in a book on AR for industrial enterprise? Because Pokémon created the “Ooooh, I get it!” moment. And that has a ripple effect.

Industrial Enterprise AR: The Inflection Point

People now understand that AR is about seeing digital things in and on top of the physical world. And since it’s understood, it’s now embedded in culture and collective consciousness. That is one of the openings AR makers have been waiting for! Fast Company wrote a piece entitled “Pokémon Go May Prove That AR Is More Mainstream Than VR.” And it’s true. Not only does the ubiquity of this seemingly simple game create an opportunity for small, local businesses—mom-and-pop shops that are willing to open themselves to it (or capitalize on the customers already coming)—but it also exposes a much larger, broader audience.

(Which means the social benefits are pretty great too. Go red team.)

More people who “get” AR means more people will get AR: in new apps, in new hardware, and in new use cases.

I do not know the people at Blue Hill Research, but they hit the nail on the head when they said, “Pokémon GO just accelerated adoption for enterprise augmented reality.” They go on to say:

Vendors such as PTC must be ecstatic about the excitement surrounding this game. As the key arms/platform dealer of augmented reality in the enterprise, PTC is in a position to translate the success of this new trend into real enterprise value through its combination of ThingWorx Studio, VuMarks [also known as ThingMarks] that can serve both as enhanced bar codes and placeholders for 3D digital representations that can be seen through a smartphone or tablet, and the professional services from the Vuforia business unit that are expert in deploying augmented reality. As odd as it sounds, ThingWorx Studio has to be seen as a leading vendor for “Pokémonifying” your business.

They have positioned themselves well at a very good time.

You Should Get Really Clear Up Front: This Is Not VR

As augmented reality has developed, it has been lumped together with virtual reality. That’s happened for a lot of reasons. Some AR systems use the same eyewear and headset tech as VR. Industry analysts who wanted to make the industry look bigger (or who didn’t yet know how to parse it out) did some creative things with numbers. The earliest makers of AR were people who came from the VR world (and that’s often still the case). So the conflation of the two is understandable.

But it is time for a separation.

AR is a distinct technology. While VR brings you into the digital world, augmented reality brings digital information into your world—and overlays it onto the physical environment around you.

People are now calling the field mixed reality,¹ which is an accurate term for the nature of what’s happening, but one that is often used as a catchall for any new tech in this arena. It effectively still lumps AR and VR together—wrong move.

Make the distinction in your mind (and business strategies) now. AR is distinct from VR. Business strategies, types of development, and financial forecasts arise depending on what technology you’re talking about. Digi-Capital, a Menlo Park–based industry analyst and consultancy, predicts that AR will be a $90 billion market by 2020—while the VR market will be worth $30 billion. The streams have just separated.

Is now really the time for AR? In industrial enterprise, it is.

The reason is precisely because of how long it’s been developing. Picture in your mind the J-curve (hockey stick) that shows the advancement of any technology. This one had a long curve. And it’s now on the upswing—and rising fast. People have been working on AR for 15 years (or 90, depending on who is counting). And all that research and thought has really paid off in the past two years or so. AR has become the beneficiary of advances in hardware, breakthroughs in computer vision technologies, and the distillation of years of thought that have gone into how to build these systems. Said more simply: the underlying technology that powers AR has finally caught up with the promise of it. As you’ll discover, AR has already quietly taken hold in industry. We’ve begun to exit the R&D phase. Now is the time for smart companies to create strategy.

Why Now Is the Best Time to Develop AR

Augmented reality represents a new way of seeing, and seeing into, the world. In this context, it is a powerful tool that not only “augments” the world around us, but—more importantly in a business context—enhances human abilities and provides greater insight into machines and products as well.

Imagine you just bought a new piece of machinery for your office or factory. Rather than ever reading an owner’s manual, you can join your entire team in “seeing” how to properly operate, maintain, and repair the machine.

Or put yourself in the shoes of a worker on an oil rig. Augmented reality can give you complex startup, shutdown, and emergency procedures in simple visual instructions, and it can connect you with immediate remote assistance should you discover a problem.

Each of these scenarios is already happening. There are systems that allow salespeople, doctors, steelworkers, electricians, and workers of all types to “augment” themselves with smartphones, tablets, and eyewear that enable them to see in the physical world the information they need to do their jobs more effectively.

A number of these use cases focus on manufacturing, safety, quality, assembly, construction, inspection, and maintenance. Because of the sheer scale of manufacturing, these are some of the most striking examples of how companies have been deploying AR. Even through pilot programs, there is evidence of deeper knowledge, significant savings (think billions of dollars), and environmental crises averted. In other words, even on a small scale, you can see the possibilities this technology opens up when rolled out (and there are more rollouts coming).

AR can also be applied to areas beyond heavy industry and manufacturing. For instance, for the last few years Mercedes Benz has been putting QR codes in the B-pillars and inside the fuel door of all its new cars. The reason: so first responders at accidents can connect with an AR app to see color-coded diagrams of wiring and fuel systems. This allows them to cut through these systems at accident sites quickly and accurately—saving lives in the process. That is the power of AR used thoughtfully.

Later in this report, you’ll hear about several industries—from aerospace to heavy machinery—where AR has been used in meaningful ways. Enterprise means business. And the business applications of AR are myriad.

The question is: Why is now the right time to invest in this (after all, hasn’t AR been around for a long time)? The answer is: Yes, it is (and yes, it has). Not only have the base technologies underlying AR changed, but the world has also changed.

Most notably in this case, we are no longer living in a world that is paper based or centered on written instruction manuals. For a lot of things that are manufactured, we already have the digital manuals, CAD models, or videos showing how they are built and assembled—which is incredibly valuable if you are building a custom AR system from scratch. To be able to show someone how to operate or repair a machine, you first have to be able to visually, accurately, graphically show the machine. We can now do that (more or less easily), drawing from the digital world. What is more, we also have adequate knowledge about how to translate these digital images into tools that can be used in different ways for enterprise AR so we can overlay and display the important bits of it in front of us. That means we can “holographically” display and render objects accurately. And we can have exactly—and only—the bits we want, exactly when we want them.

The barrier to entry is lower than it ever has been. Computer vision systems have improved. Everyone has a tablet or smartphone. And now, for the first time, there are mature consultancies that specialize in enterprise AR (and have actually applied it), and enterprise-focused platforms that help industrial clients create and roll out AR apps and programs. The requisite systems are in place, and the tools needed to prototype a system are less expensive as well.

Knowing more about the world around you, in real time, without having to look that information up yourself is the equivalent of having x-ray vision and a genie in your pocket who makes the exact information you need appear exactly when you need it.

In his book The Road Ahead, Bill Gates said, “We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten.”

AR Has Reached the Usability Phase

We’ve passed the imagination phase of the 1900s–1960s. We’ve completed the R&D phase of the 90s. Inexpensive tablets, improved computer vision systems, and people who have been playing with the technology long enough to advance it significantly have now put us squarely at the front end of the “finally usable by a broad group of humans” phase. It’s still early in that era...it is dawning now. And, by doing your own research, prototyping and planning, and investing, you will put yourself in a great competitive position as this evolves. Not everyone needs to be an expert at building or authoring AR systems. But everyone does need to have a basic understanding of how AR systems are evolving, so you’ll know how to use them to your advantage. Learning now is a smart business move.

In 2016, CB Insights presented at a 500 Startups conference called “PreMoney.” The event and the presentation were designed for venture capitalists and outlined the hottest areas of investment—and AR was one of them. The crux of the presentation was very much along the same lines of what Technicolor Ventures’ Mark Linao wrote in a guest post for the CB Insights blog: “Despite the dominance of gaming in media coverage of VR/AR, it’s actually commercial and industrial applications that take the most deals. The number of VR/AR startups is increasing as venture capital dollars continue to flood in, with investment reaching an all-time high in 2015 of $695 million in equity funding across 126 deals, according to CB Insights data.” Or course, Magic Leap made up a big chunk of that. But drilling deeper into the funding data, Linao said, “commercial and industrial applications received approximately 17% of total investments among all categories.” What that means: AR is steadily—if slowly—being adopted by the enterprise. And it is being noticed by venture capitalists and big investors.

So two answers to “why now?” are because the technology has significantly advanced and because other smart people are doing it. But by looking closely for yourself at the way industry is changing, you may find even deeper business reasons.

Where AR and IoT Connect

Not only is the history of AR-enabling devices like tablets and smartphones and wearable systems important to discuss, but there’s an evolution of sensors that is relevant too.

The kinds of sensors we have in modern consumer devices were once so incredibly expensive that only NASA could afford to invent (and build) them. If you look around the world today, everything is sensor-enabled or controlled. Elevators, pressurized doors in skyscrapers, and almost all factories are filled with sensors. If you walk a factory floor, you might have dozens of sensors on a single machine that are tracking everything from vibration of multiple axes to pressures to temperatures to mass flow—and you’ll also have variable-frequency drives controlling the motor. In addition, there is an explosion of wireless sensors in the marketplace, all less expensive and smaller than ever. Why is that relevant?

We’ve entered the era of the industrial Internet of Things (IoT), which is known by a lot of different names, depending on what part of the world you are in. Some people—mostly in Germany—call it Industry 4.0. Some people refer to it as smart, connected products. People have also used the phrase Brilliant Factory, or even the 4th Industrial Revolution. But when you distill it down, what they’re trying to communicate is the connection between machines, data, and people.

With industrialized, instrumented machines, you also have sensors. More sensors on these machines are pulling more data off of them. Since all of these machines have sensors, and all of them are pulling data, that’s a lot of information. What you want to do is take the data and analytics and push the most meaningful information out to the workers, to give them data visualization so they can do their jobs better. Once they get that information, they act on it with the machine, and it becomes a continuous improvement cycle.

That’s what AR does. AR enables the visualization and the contextualization of information. AR allows you to translate the data about your machines, within your machines, and coming from your machines in a way that’s actually useful.

Glen Fields from PTC puts it this way:

I came up through the marine industry on ships. It was important to be in the control room and to monitor the condition of the hundreds and hundreds of different machines that were all contributing to the propulsion of the ship. But if you just strictly stayed in the control room and monitored and relied on all of the gauges and the sensors, you didn’t gain an appreciation for what was really happening out on the machine room floor. You might have machines that are smoking and the sensors might be bad. You want to see what’s really going on.

With AR, we’re trying to take the control room out of the control room.

So you have the Internet of Things and you have augmented reality. But how do the two work together? The IoT is essentially the ability to digitally talk to physical things—to monitor and to manage them. AR is all about the ability to see and experience the digital attributes of those physical things.

The ability to deliver digital information in the real world, in context with the machine, is what AR makes possible. It enhances that whole technology stack.

Right now, you have hardware. You have the software that’s embedded into that physical product, and you have communications that allow you to convey what’s happening digitally with that machine. The data on what’s happening in the machine can be in the physical world, or it can be in the cloud in the form of databases and analytics and platforms and applications.

Because AR also enables video communication and video augmentation, it allows you to experience that hardware data in a totally different way. Brains don’t just process information in two dimensions. Screens are great, but they are not the same as interacting with a physical object the way we typically do. That physical connection is one key reason augmented reality of the enterprise is gaining traction.

Workforce Transformation: AR for Knowledge Transfer (Industries Are Blending)

In addition to helping to visualize and contextualize information, AR also helps smooth a social issue facing all world economies: the aging workforce and shortage of skilled manufacturing labor. It’s an important area to consider. The latest data shared publicly on the state of global manufacturing, a joint study by Deloitte and the World Economic Forum, says that the manufacturing industry can expect a shortage of 10 million workers globally. More recent reports say that 2 million of those are in the United States. What you’re seeing are baby boomers retiring, a dwindling Chinese workforce, and students coming out of school today that don’t have the same skills that a 30-year veteran has. The average age for a highly skilled manufacturing worker in the United States today is 56. That’s the average age. As these skilled workers retire, it’s going to create a gap in the workforce.

Why is that relevant? Augmented reality provides an ability to transfer that knowledge, even to unskilled workers. It enables you to collect and preserve the information that lives in the heads of these highly skilled workers and digitally capture it in a few ways:

Augmented instructions

AR can be in the form of service. By gathering information from a skilled worker, you can develop augmented work instructions about how to do a complex repair on a piece of equipment. You can easily use that in training and simulation—teaching people to do jobs before they go into the workforce.

Remote guides

AR can also allow skilled workers to act as remote guides for less skilled workers. This creates the possibility for “retired” workers to reenter the workforce, part-time from anywhere. It also makes it possible for one highly skilled employee to have “many hands,” working through less skilled workers (using AR systems and visuals as guides) to multiply value. It is now possible to be in many places at one time.

Cross-Domain Technical Skills

Industries are blending. Technology is one of the universal bridges between all of them, and AR can strengthen that bridge. A May 2016 study by the World Economic Forum, called “Manufacturing Our Future,” says:

The future of manufacturing will see an increased demand for cross-domain skills covering technology, engineering, electronics, robotics, usage of new equipment, computational thinking, coding and computer sciences...The typical blue- vs. white-collar job distinction does not hold anymore. While former white-collar workers have moved up into innovation, former blue-collar workers are required to perform more capability and cross-domain-based functions in manufacturing, as machines take over manual jobs.

This competitive advantage is recognized by all sorts of people. At the end of last year, Fast Company reported on a special project spearheaded by the president of Kazakhstan. He invited multiple companies, including Intel’s Daqri team, to come to Kazakhstan to visit a facility called KSP Steel, to show how technology could improve their society, and to educate their new workforce. Specifically, the Daqri team showed the workers at this steel plant how to use AR to enhance production and make their jobs easier.

You can read the public version of the case study yourself. The results were remarkable. KSP Steel was able to improve the productivity of its workers by 40% and decrease downtime at the facility by 50% using digital information in context with the real world.

What Is the Business Justification for This (AKA Show Me the Numbers)?

So how much return could there be for you? How can you start to think about this if you are trying to create a business justification?

I interviewed Matt Sheridan at PTC to find an answer. What he said was brilliant:

What we put together is a way you can think about it, and a way that you can start to have the conversation in your company. The example we [focus on] is service. There’s a study that was by a partner of ours called ServiceMax. The study is called “A Diamond In the Rough: Unleashing the Power of Field Service Transformation.”

The ServiceMax study talks about two aspects of service. First, that service is continuing to grow—in the sense that there’s more service taking place—and various companies are looking at service as a revenue stream. Before, service was simply the add-on for selling a product. Now companies are actually selling the service itself as part of their offering. Therefore revenue is increasing.

The study says, “In 2012 there were about 5.4 million service technicians. By 2020 there will be a need for 6.2 million.” That means nearly a million more technicians in the workforce. When you tie that back to the aging workforce and add the fact that it’s a challenge to find skilled labor, the conclusion is clear: this is going to be something companies really struggle with.

Sheridan also explains:

If you look at your service department, most good service organizations have some metric that they’re tracking. First-time fix, or mean time to repair, or mean time to install. If you were going to have a conversation about how AR might improve a particular area and you wanted to tie that to something, let’s take a look at first-time fix and out-of-service time, what do we do?

Referring back to that ServiceMax whitepaper they wrote, it talks specifically about service of two companies. Company A has a first-time fix rate of 88%—considered at the high end of the first-time fix rate range. Company B was at 10%, at the low end of the first-time fix rate. So if both companies are doing 400 service jobs a day, the first-time fix rate differentiation means Company A is able to fit in 100 more jobs a day. Multiply that by 250 days a year, that’s 25,000 extra jobs. So for Company B to compete with Company A, they need roughly 25 more people to complete all those jobs. I then looked at Payscale.com and looked up the average salary of a service technician in North America and found it to be $55,000 a year. Simple math: 25 times $55,000 a year gives you $1.3 million investment for Company B to catch up with Company A. That gives you a ballpark number.

What that means: if you find what’s causing the first-time fix rate problems, and what’s causing increased cost, then you can tie that to how AR can help solve the problem.

“It’s a little bit of A equals B, B equals C, and therefore A equals C,” Sheridan concludes. “We don’t have at the moment the exact company that says, ‘I saved $1.3 million with AR.’ But you can see where the conversations should be happening. And people like real numbers, so they’ll have those conversations.”

What Else Should You Know?

After seeing how AR is developing, you might be asking the next big question: How is it successfully being applied in an industrial enterprise context?

Chapter 2, AR Creators and Use Cases You Should Know outlines case studies that answer just that.

¹ For more about mixed reality and why it is a term you should pay close attention to, see Appendix A.