book

Creating Augmented and Virtual Realities

by Erin Pangilinan, Steve Lukas, Vasanth Mohan

April 2019

Intermediate to advanced

370 pages

10h 23m

English

O'Reilly Media, Inc.

Read now

Unlock full access

A Note on TerminologyWhy We Wrote This BookWhat We Didn’t Cover in This BookHow This Book Is OrganizedConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Common Term DefinitionIntroductionModalities Through the Ages: Pre-Twentieth CenturyModalities Through the Ages: Through World War IIModalities Through the Ages: Post-World War IIModalities Through the Ages: The Rise of Personal ComputingModalities Through the Ages: Computer MiniaturizationWhy Did We Just Go Over All of This?Types of Common HCI ModalitiesNew ModalitiesThe Current State of Modalities for Spatial Computing DevicesCurrent Controllers for Immersive Computing SystemsBody Tracking TechnologiesA Note on Hand Tracking and Hand Pose RecognitionVoice, Hands, and Hardware Inputs over the Next Generation
Envisioning a FutureSensory Technology ExplainedSo, Who Are We Building This Future For?The Role of Women in AISensory DesignAn IntroductionFive Sensory Principles1. Intuitive Experiences Are Multisensory2. 3D Will Be Normcore3. Designs Become Physical Nature4. Design for the Uncontrollable5. Unlock the Power of Spatial CollaborationAdobe’s AR StoryConclusion
A More Natural Way of Making 3D ArtVR for Animation
IntroductionOptions to ConsiderIdeal SolutionTopologyBakingDraw CallsUsing VR Tools for Creating 3D ArtAcquiring 3D Models Versus Making Them from ScratchSummary
Who Are We?A Brief History of ARHow and Why to Select an AR PlatformI’m a Developer, What Platform Should I Use and Why?Performance Is StatisticsIntegrating Hardware and SoftwareOptical CalibrationInertial CalibrationThe Future of TrackingThe Future of AR Computer VisionMappingHow Does Multiplayer AR Work?What’s the Difficult Part?How Does Relocalization Work?What’s the State of the Art in Research (and Coming Soon to Consumer)?Can the Relocalization Problem Really Be Solved for Consumers?Can’t Google or Apple Just Do This?Relocalization != Multiplayer; It’s Also Critical for…How Is Relocalization Really Being Done Today in Apps?PlatformsApple’s ARKitSome Mysteries ExplainedIsn’t ARCore Just Tango-Lite?So, Should I Build on ARCore Now?What About Tango, Hololens, Vuforia, and Others?Other Development ConsiderationsLightingMultiplayer AR—Why It’s Quite DifficultHow Do People Connect Through AR?How Do AR Apps Connect to the World and Know Where They Really Are?How Do AR Apps Understand and Connect to Things in the Real World?The AR CloudWhat I Envision When I Think About the AR CloudHow Big a Deal Is the AR Cloud?So, Just What Is This AR Cloud?Why All the “Meh” from the Press for ARKit and ARCore?What’s Missing to Make a Great AR App?Is Today’s Mobile Cloud up to the Job?Is ARKit (or ARCore) Useless Without the AR Cloud?The Dawn of the AR CloudThe Bigger Picture—Privacy and AR Cloud DataGlossary

Why Cross-Platform?The Role of Game EnginesUnderstanding 3D GraphicsThe Virtual CameraDegrees of FreedomPortability Lessons from Video Game DesignSimplifying the Controller InputDevelopment Step 1: Designing the Base InterfaceDevelopment Step 2: Platform IntegrationSummary
What Is VRTK and Why People Use It?The History of VRTKWelcome to the SteamVR Unity ToolkitVRTK v4The Future of VRTKThe Success of VRTKGetting Started with VRTK 4
Developing for Virtual Reality and Augmented Reality Is DifficultHandling LocomotionLocomotion in VRLocomotion in AREffective Use of AudioAudio in VRAudio in ARCommon Interactions ParadigmsInventory for VRAugmented Reality RaycastsConclusion
IntroductionUnderstanding Data VisualizationPrinciples for Data and Machine Learning Visualization in Spatial ComputingWhy Data and Machine Learning Visualization Works in Spatial ComputingThe Evolution of Data Visualization Design with the Emergence of XR2D and 3D Data Represented in XR2D Data Visualizations versus 3D Data Visualization in Spatial ComputingInteractivity in Data Visualizations in Spatial ComputingAnimationFailures in Data Visualization DesignGood Data Visualization Design Optimize 3D Spaces“Savings in Time Feel Like Simplicity”Data Representations, Infographics, and InteractionsWhat Qualifies as Data Visualization?Defining Distinctions in Data Visualization and Big Data or Machine Learning VisualizationsHow to Create Data Visualization: Data Visualization Creation PipelineWebXR: Building Data Visualizations for the WebData Visualization Challenges in XRData Visualization Industry Use Case Examples of Data Visualizations3D Reconstruction and Direct Manipulation of Real-World Data: Anatomical Structures in XRA Closer Look at Glass BrainTVA Surg Medical Imaging VR ModuleData Visualization Is for Everyone: Open Source–Based Data Visualization in XRProtein Data VisualizationHands-On Tutorials: How to Create Data Visualization in Spatial ComputingHow to Create Data Visualization: ResourcesConclusionReferencesResourcesData Visualization ToolsMachine Learning Visualization ToolsData Journalism VisualizationsGlossary
IntroductionBehaviorsCurrent Practice: Reactive AIAdaptabilityComplexity and UniversalityFeasibilityMore Intelligence in the System: Deliberative AIMachine LearningReinforcement LearningDeep Reinforcement LearningImitation LearningCombining Automated Planning and Machine LearningApplicationsConclusionReferences
VR/AR Health Technology Application DesignStandard UX Isn’t IntuitivePick a Calm EnvironmentConvenienceTutorial: Insight Parkinson’s ExperimentWhat Insight DoesHow It Was BuiltCompaniesPlanning and GuidanceExperiences Designed for Medical EducationExperiences Designed for Use by PatientsProactive HealthCase Studies from Leading Academic Institutions
IntroductionPart 1: Five Key Principles of AR and VR for SportsNothing Is LivePart 2: The Next Evolution of Sports ExperiencesPart 3: Making the FutureOwnershipFinal ThoughtConclusion
Introduction: The Importance of Enterprise TrainingDoes VR Training Work?Use Case: Flood House TrainingWhat Is VR Training Good for? R.I.D.E.What Makes Good VR Training?Spherical VideoThe Benefits of Spherical VideoThe Challenges of Spherical VideoInteractions with Spherical VideoUse Case: Factory Floor TrainingThe Role of NarrativeUse Case: Store Robbery TrainingThe Future of XR Training: Beyond Spherical VideoComputer GraphicsUse Case: Soft Skills TrainingThe Future: PhotogrammetryThe Future: Light FieldsThe Future: AR TrainingThe Future: Voice RecognitionThe Future: The Ideal Training ScenarioReferences

Content preview from Creating Augmented and Virtual Realities

Part IV. Creating Cross-Platform Augmented Reality and Virtual Reality

When Facebook bought Oculus Rift in 2014 for $2 billion, the investment industry became catalyzed with virtual reality (VR) validated as ushering in the dawn of a new age of computing. Slowly the market became inundated with Oculus alternatives in the war for the space on your face. Until that point, augmented reality (AR) was practically dominated by Vuforia, an image-based tracking solution. But with the Oculus acquisition, suddenly several companies and products in the immersive space quickly rose to recognition: Meta and Magic Leap for AR, Samsung GearVR and Google Cardboard, and then Hololens and Daydream among many others.

As an investor with Qualcomm Ventures, Steve Lukas was tasked with finding which companies would “win” in the VR/AR space; that’s when a massive problem was identified: a software ecosystem that was limited and fracturing further with each headset release. This wasn’t fully clear until 2016, when all of the platforms began releasing to the public. As is the usual case with a new industry, the most vocal proponents of VR caused an over-expectation of adoption uptake, and VR ended up selling in lower numbers than originally anticipated. This was consistent for every headset release, with each platform looking like it would be the one that would take the market mainstream, whether it was the full power of HTC Vive, the free Samsung GearVRs distributed with their new phones, the low cost ...