book

Creating Augmented and Virtual Realities

by Erin Pangilinan, Steve Lukas, Vasanth Mohan

April 2019

Intermediate to advanced

370 pages

10h 23m

English

O'Reilly Media, Inc.

Read now

Unlock full access

A Note on TerminologyWhy We Wrote This BookWhat We Didn’t Cover in This BookHow This Book Is OrganizedConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Common Term DefinitionIntroductionModalities Through the Ages: Pre-Twentieth CenturyModalities Through the Ages: Through World War IIModalities Through the Ages: Post-World War IIModalities Through the Ages: The Rise of Personal ComputingModalities Through the Ages: Computer MiniaturizationWhy Did We Just Go Over All of This?Types of Common HCI ModalitiesNew ModalitiesThe Current State of Modalities for Spatial Computing DevicesCurrent Controllers for Immersive Computing SystemsBody Tracking TechnologiesA Note on Hand Tracking and Hand Pose RecognitionVoice, Hands, and Hardware Inputs over the Next Generation
Envisioning a FutureSensory Technology ExplainedSo, Who Are We Building This Future For?The Role of Women in AISensory DesignAn IntroductionFive Sensory Principles1. Intuitive Experiences Are Multisensory2. 3D Will Be Normcore3. Designs Become Physical Nature4. Design for the Uncontrollable5. Unlock the Power of Spatial CollaborationAdobe’s AR StoryConclusion
A More Natural Way of Making 3D ArtVR for Animation
IntroductionOptions to ConsiderIdeal SolutionTopologyBakingDraw CallsUsing VR Tools for Creating 3D ArtAcquiring 3D Models Versus Making Them from ScratchSummary
Who Are We?A Brief History of ARHow and Why to Select an AR PlatformI’m a Developer, What Platform Should I Use and Why?Performance Is StatisticsIntegrating Hardware and SoftwareOptical CalibrationInertial CalibrationThe Future of TrackingThe Future of AR Computer VisionMappingHow Does Multiplayer AR Work?What’s the Difficult Part?How Does Relocalization Work?What’s the State of the Art in Research (and Coming Soon to Consumer)?Can the Relocalization Problem Really Be Solved for Consumers?Can’t Google or Apple Just Do This?Relocalization != Multiplayer; It’s Also Critical for…How Is Relocalization Really Being Done Today in Apps?PlatformsApple’s ARKitSome Mysteries ExplainedIsn’t ARCore Just Tango-Lite?So, Should I Build on ARCore Now?What About Tango, Hololens, Vuforia, and Others?Other Development ConsiderationsLightingMultiplayer AR—Why It’s Quite DifficultHow Do People Connect Through AR?How Do AR Apps Connect to the World and Know Where They Really Are?How Do AR Apps Understand and Connect to Things in the Real World?The AR CloudWhat I Envision When I Think About the AR CloudHow Big a Deal Is the AR Cloud?So, Just What Is This AR Cloud?Why All the “Meh” from the Press for ARKit and ARCore?What’s Missing to Make a Great AR App?Is Today’s Mobile Cloud up to the Job?Is ARKit (or ARCore) Useless Without the AR Cloud?The Dawn of the AR CloudThe Bigger Picture—Privacy and AR Cloud DataGlossary

Why Cross-Platform?The Role of Game EnginesUnderstanding 3D GraphicsThe Virtual CameraDegrees of FreedomPortability Lessons from Video Game DesignSimplifying the Controller InputDevelopment Step 1: Designing the Base InterfaceDevelopment Step 2: Platform IntegrationSummary
What Is VRTK and Why People Use It?The History of VRTKWelcome to the SteamVR Unity ToolkitVRTK v4The Future of VRTKThe Success of VRTKGetting Started with VRTK 4
Developing for Virtual Reality and Augmented Reality Is DifficultHandling LocomotionLocomotion in VRLocomotion in AREffective Use of AudioAudio in VRAudio in ARCommon Interactions ParadigmsInventory for VRAugmented Reality RaycastsConclusion
IntroductionUnderstanding Data VisualizationPrinciples for Data and Machine Learning Visualization in Spatial ComputingWhy Data and Machine Learning Visualization Works in Spatial ComputingThe Evolution of Data Visualization Design with the Emergence of XR2D and 3D Data Represented in XR2D Data Visualizations versus 3D Data Visualization in Spatial ComputingInteractivity in Data Visualizations in Spatial ComputingAnimationFailures in Data Visualization DesignGood Data Visualization Design Optimize 3D Spaces“Savings in Time Feel Like Simplicity”Data Representations, Infographics, and InteractionsWhat Qualifies as Data Visualization?Defining Distinctions in Data Visualization and Big Data or Machine Learning VisualizationsHow to Create Data Visualization: Data Visualization Creation PipelineWebXR: Building Data Visualizations for the WebData Visualization Challenges in XRData Visualization Industry Use Case Examples of Data Visualizations3D Reconstruction and Direct Manipulation of Real-World Data: Anatomical Structures in XRA Closer Look at Glass BrainTVA Surg Medical Imaging VR ModuleData Visualization Is for Everyone: Open Source–Based Data Visualization in XRProtein Data VisualizationHands-On Tutorials: How to Create Data Visualization in Spatial ComputingHow to Create Data Visualization: ResourcesConclusionReferencesResourcesData Visualization ToolsMachine Learning Visualization ToolsData Journalism VisualizationsGlossary
IntroductionBehaviorsCurrent Practice: Reactive AIAdaptabilityComplexity and UniversalityFeasibilityMore Intelligence in the System: Deliberative AIMachine LearningReinforcement LearningDeep Reinforcement LearningImitation LearningCombining Automated Planning and Machine LearningApplicationsConclusionReferences
VR/AR Health Technology Application DesignStandard UX Isn’t IntuitivePick a Calm EnvironmentConvenienceTutorial: Insight Parkinson’s ExperimentWhat Insight DoesHow It Was BuiltCompaniesPlanning and GuidanceExperiences Designed for Medical EducationExperiences Designed for Use by PatientsProactive HealthCase Studies from Leading Academic Institutions
IntroductionPart 1: Five Key Principles of AR and VR for SportsNothing Is LivePart 2: The Next Evolution of Sports ExperiencesPart 3: Making the FutureOwnershipFinal ThoughtConclusion
Introduction: The Importance of Enterprise TrainingDoes VR Training Work?Use Case: Flood House TrainingWhat Is VR Training Good for? R.I.D.E.What Makes Good VR Training?Spherical VideoThe Benefits of Spherical VideoThe Challenges of Spherical VideoInteractions with Spherical VideoUse Case: Factory Floor TrainingThe Role of NarrativeUse Case: Store Robbery TrainingThe Future of XR Training: Beyond Spherical VideoComputer GraphicsUse Case: Soft Skills TrainingThe Future: PhotogrammetryThe Future: Light FieldsThe Future: AR TrainingThe Future: Voice RecognitionThe Future: The Ideal Training ScenarioReferences

Overview

Despite popular forays into augmented and virtual reality in recent years, spatial computing still sits on the cusp of mainstream use. Developers, artists, and designers looking to enter this field today have few places to turn for expert guidance. In this book, Erin Pangilinan, Steve Lukas, and Vasanth Mohan examine the AR and VR development pipeline and provide hands-on practice to help you hone your skills.

Through step-by-step tutorials, you’ll learn how to build practical applications and experiences grounded in theory and backed by industry use cases. In each section of the book, industry specialists, including Timoni West, Victor Prisacariu, and Nicolas Meuleau, join the authors to explain the technology behind spatial computing.

In three parts, this book covers:

Art and design: Explore spatial computing and design interactions, human-centered interaction and sensory design, and content creation tools for digital art
Technical development: Examine differences between ARKit, ARCore, and spatial mapping-based systems; learn approaches to cross-platform development on head-mounted displays
Use cases: Learn how data and machine learning visualization and AI work in spatial computing, training, sports, health, and other enterprise applications