book

Creating Augmented and Virtual Realities

by Erin Pangilinan, Steve Lukas, Vasanth Mohan

April 2019

Intermediate to advanced

370 pages

10h 23m

English

O'Reilly Media, Inc.

Read now

Unlock full access

A Note on TerminologyWhy We Wrote This BookWhat We Didn’t Cover in This BookHow This Book Is OrganizedConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Common Term DefinitionIntroductionModalities Through the Ages: Pre-Twentieth CenturyModalities Through the Ages: Through World War IIModalities Through the Ages: Post-World War IIModalities Through the Ages: The Rise of Personal ComputingModalities Through the Ages: Computer MiniaturizationWhy Did We Just Go Over All of This?Types of Common HCI ModalitiesNew ModalitiesThe Current State of Modalities for Spatial Computing DevicesCurrent Controllers for Immersive Computing SystemsBody Tracking TechnologiesA Note on Hand Tracking and Hand Pose RecognitionVoice, Hands, and Hardware Inputs over the Next Generation
Envisioning a FutureSensory Technology ExplainedSo, Who Are We Building This Future For?The Role of Women in AISensory DesignAn IntroductionFive Sensory Principles1. Intuitive Experiences Are Multisensory2. 3D Will Be Normcore3. Designs Become Physical Nature4. Design for the Uncontrollable5. Unlock the Power of Spatial CollaborationAdobe’s AR StoryConclusion
A More Natural Way of Making 3D ArtVR for Animation
IntroductionOptions to ConsiderIdeal SolutionTopologyBakingDraw CallsUsing VR Tools for Creating 3D ArtAcquiring 3D Models Versus Making Them from ScratchSummary
Who Are We?A Brief History of ARHow and Why to Select an AR PlatformI’m a Developer, What Platform Should I Use and Why?Performance Is StatisticsIntegrating Hardware and SoftwareOptical CalibrationInertial CalibrationThe Future of TrackingThe Future of AR Computer VisionMappingHow Does Multiplayer AR Work?What’s the Difficult Part?How Does Relocalization Work?What’s the State of the Art in Research (and Coming Soon to Consumer)?Can the Relocalization Problem Really Be Solved for Consumers?Can’t Google or Apple Just Do This?Relocalization != Multiplayer; It’s Also Critical for…How Is Relocalization Really Being Done Today in Apps?PlatformsApple’s ARKitSome Mysteries ExplainedIsn’t ARCore Just Tango-Lite?So, Should I Build on ARCore Now?What About Tango, Hololens, Vuforia, and Others?Other Development ConsiderationsLightingMultiplayer AR—Why It’s Quite DifficultHow Do People Connect Through AR?How Do AR Apps Connect to the World and Know Where They Really Are?How Do AR Apps Understand and Connect to Things in the Real World?The AR CloudWhat I Envision When I Think About the AR CloudHow Big a Deal Is the AR Cloud?So, Just What Is This AR Cloud?Why All the “Meh” from the Press for ARKit and ARCore?What’s Missing to Make a Great AR App?Is Today’s Mobile Cloud up to the Job?Is ARKit (or ARCore) Useless Without the AR Cloud?The Dawn of the AR CloudThe Bigger Picture—Privacy and AR Cloud DataGlossary

Why Cross-Platform?The Role of Game EnginesUnderstanding 3D GraphicsThe Virtual CameraDegrees of FreedomPortability Lessons from Video Game DesignSimplifying the Controller InputDevelopment Step 1: Designing the Base InterfaceDevelopment Step 2: Platform IntegrationSummary
What Is VRTK and Why People Use It?The History of VRTKWelcome to the SteamVR Unity ToolkitVRTK v4The Future of VRTKThe Success of VRTKGetting Started with VRTK 4
Developing for Virtual Reality and Augmented Reality Is DifficultHandling LocomotionLocomotion in VRLocomotion in AREffective Use of AudioAudio in VRAudio in ARCommon Interactions ParadigmsInventory for VRAugmented Reality RaycastsConclusion
IntroductionUnderstanding Data VisualizationPrinciples for Data and Machine Learning Visualization in Spatial ComputingWhy Data and Machine Learning Visualization Works in Spatial ComputingThe Evolution of Data Visualization Design with the Emergence of XR2D and 3D Data Represented in XR2D Data Visualizations versus 3D Data Visualization in Spatial ComputingInteractivity in Data Visualizations in Spatial ComputingAnimationFailures in Data Visualization DesignGood Data Visualization Design Optimize 3D Spaces“Savings in Time Feel Like Simplicity”Data Representations, Infographics, and InteractionsWhat Qualifies as Data Visualization?Defining Distinctions in Data Visualization and Big Data or Machine Learning VisualizationsHow to Create Data Visualization: Data Visualization Creation PipelineWebXR: Building Data Visualizations for the WebData Visualization Challenges in XRData Visualization Industry Use Case Examples of Data Visualizations3D Reconstruction and Direct Manipulation of Real-World Data: Anatomical Structures in XRA Closer Look at Glass BrainTVA Surg Medical Imaging VR ModuleData Visualization Is for Everyone: Open Source–Based Data Visualization in XRProtein Data VisualizationHands-On Tutorials: How to Create Data Visualization in Spatial ComputingHow to Create Data Visualization: ResourcesConclusionReferencesResourcesData Visualization ToolsMachine Learning Visualization ToolsData Journalism VisualizationsGlossary
IntroductionBehaviorsCurrent Practice: Reactive AIAdaptabilityComplexity and UniversalityFeasibilityMore Intelligence in the System: Deliberative AIMachine LearningReinforcement LearningDeep Reinforcement LearningImitation LearningCombining Automated Planning and Machine LearningApplicationsConclusionReferences
VR/AR Health Technology Application DesignStandard UX Isn’t IntuitivePick a Calm EnvironmentConvenienceTutorial: Insight Parkinson’s ExperimentWhat Insight DoesHow It Was BuiltCompaniesPlanning and GuidanceExperiences Designed for Medical EducationExperiences Designed for Use by PatientsProactive HealthCase Studies from Leading Academic Institutions
IntroductionPart 1: Five Key Principles of AR and VR for SportsNothing Is LivePart 2: The Next Evolution of Sports ExperiencesPart 3: Making the FutureOwnershipFinal ThoughtConclusion
Introduction: The Importance of Enterprise TrainingDoes VR Training Work?Use Case: Flood House TrainingWhat Is VR Training Good for? R.I.D.E.What Makes Good VR Training?Spherical VideoThe Benefits of Spherical VideoThe Challenges of Spherical VideoInteractions with Spherical VideoUse Case: Factory Floor TrainingThe Role of NarrativeUse Case: Store Robbery TrainingThe Future of XR Training: Beyond Spherical VideoComputer GraphicsUse Case: Soft Skills TrainingThe Future: PhotogrammetryThe Future: Light FieldsThe Future: AR TrainingThe Future: Voice RecognitionThe Future: The Ideal Training ScenarioReferences

Content preview from Creating Augmented and Virtual Realities

Chapter 5. How the Computer Vision That Makes Augmented Reality Possible Works

Victor Prisacariu and Matt Miesnieks

Who Are We?

My name is Matt Miesnieks, and I’m the CEO of a startup called 6D.ai, which is a spinoff of the Oxford University Active Vision Lab, where my cofounder, Professor Victor Prisacariu, supervises one of the world’s best AR computer vision research groups. I’ve spent 10 years working on AR, as a founder (Dekko), investor (Super Ventures), and executive (Samsung, Layar). I have an extensive background in software infrastructure in smartphones (Openwave) and Wireline (Ascend Communications), as an engineer and sales vice president.

At 6D.ai, we are thinking slightly differently to everyone else about AR. We are solving the most difficult technical problems and exposing the solutions via developer APIs for customers with the most challenging AR problems. We are independent and cross-platform, and we sell usage of our APIs, not advertising based on our customer data. We believe that persistence is foundational, and you can’t have persistence without treating privacy seriously. And to treat privacy seriously, it means that personally identifiable information (PII) cannot leave the device (unless explicitly allowed by the user). This creates a much more difficult technical problem to solve because it means building and searching a large SLAM map on-device, and in real time. This is technically easy-ish to do with small maps and anchors, but it’s very, very difficult ...