book

Creating Augmented and Virtual Realities

by Erin Pangilinan, Steve Lukas, Vasanth Mohan

April 2019

Intermediate to advanced

370 pages

10h 23m

English

O'Reilly Media, Inc.

Read now

Unlock full access

A Note on TerminologyWhy We Wrote This BookWhat We Didn’t Cover in This BookHow This Book Is OrganizedConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Common Term DefinitionIntroductionModalities Through the Ages: Pre-Twentieth CenturyModalities Through the Ages: Through World War IIModalities Through the Ages: Post-World War IIModalities Through the Ages: The Rise of Personal ComputingModalities Through the Ages: Computer MiniaturizationWhy Did We Just Go Over All of This?Types of Common HCI ModalitiesNew ModalitiesThe Current State of Modalities for Spatial Computing DevicesCurrent Controllers for Immersive Computing SystemsBody Tracking TechnologiesA Note on Hand Tracking and Hand Pose RecognitionVoice, Hands, and Hardware Inputs over the Next Generation
Envisioning a FutureSensory Technology ExplainedSo, Who Are We Building This Future For?The Role of Women in AISensory DesignAn IntroductionFive Sensory Principles1. Intuitive Experiences Are Multisensory2. 3D Will Be Normcore3. Designs Become Physical Nature4. Design for the Uncontrollable5. Unlock the Power of Spatial CollaborationAdobe’s AR StoryConclusion
A More Natural Way of Making 3D ArtVR for Animation
IntroductionOptions to ConsiderIdeal SolutionTopologyBakingDraw CallsUsing VR Tools for Creating 3D ArtAcquiring 3D Models Versus Making Them from ScratchSummary
Who Are We?A Brief History of ARHow and Why to Select an AR PlatformI’m a Developer, What Platform Should I Use and Why?Performance Is StatisticsIntegrating Hardware and SoftwareOptical CalibrationInertial CalibrationThe Future of TrackingThe Future of AR Computer VisionMappingHow Does Multiplayer AR Work?What’s the Difficult Part?How Does Relocalization Work?What’s the State of the Art in Research (and Coming Soon to Consumer)?Can the Relocalization Problem Really Be Solved for Consumers?Can’t Google or Apple Just Do This?Relocalization != Multiplayer; It’s Also Critical for…How Is Relocalization Really Being Done Today in Apps?PlatformsApple’s ARKitSome Mysteries ExplainedIsn’t ARCore Just Tango-Lite?So, Should I Build on ARCore Now?What About Tango, Hololens, Vuforia, and Others?Other Development ConsiderationsLightingMultiplayer AR—Why It’s Quite DifficultHow Do People Connect Through AR?How Do AR Apps Connect to the World and Know Where They Really Are?How Do AR Apps Understand and Connect to Things in the Real World?The AR CloudWhat I Envision When I Think About the AR CloudHow Big a Deal Is the AR Cloud?So, Just What Is This AR Cloud?Why All the “Meh” from the Press for ARKit and ARCore?What’s Missing to Make a Great AR App?Is Today’s Mobile Cloud up to the Job?Is ARKit (or ARCore) Useless Without the AR Cloud?The Dawn of the AR CloudThe Bigger Picture—Privacy and AR Cloud DataGlossary

Why Cross-Platform?The Role of Game EnginesUnderstanding 3D GraphicsThe Virtual CameraDegrees of FreedomPortability Lessons from Video Game DesignSimplifying the Controller InputDevelopment Step 1: Designing the Base InterfaceDevelopment Step 2: Platform IntegrationSummary
What Is VRTK and Why People Use It?The History of VRTKWelcome to the SteamVR Unity ToolkitVRTK v4The Future of VRTKThe Success of VRTKGetting Started with VRTK 4
Developing for Virtual Reality and Augmented Reality Is DifficultHandling LocomotionLocomotion in VRLocomotion in AREffective Use of AudioAudio in VRAudio in ARCommon Interactions ParadigmsInventory for VRAugmented Reality RaycastsConclusion
IntroductionUnderstanding Data VisualizationPrinciples for Data and Machine Learning Visualization in Spatial ComputingWhy Data and Machine Learning Visualization Works in Spatial ComputingThe Evolution of Data Visualization Design with the Emergence of XR2D and 3D Data Represented in XR2D Data Visualizations versus 3D Data Visualization in Spatial ComputingInteractivity in Data Visualizations in Spatial ComputingAnimationFailures in Data Visualization DesignGood Data Visualization Design Optimize 3D Spaces“Savings in Time Feel Like Simplicity”Data Representations, Infographics, and InteractionsWhat Qualifies as Data Visualization?Defining Distinctions in Data Visualization and Big Data or Machine Learning VisualizationsHow to Create Data Visualization: Data Visualization Creation PipelineWebXR: Building Data Visualizations for the WebData Visualization Challenges in XRData Visualization Industry Use Case Examples of Data Visualizations3D Reconstruction and Direct Manipulation of Real-World Data: Anatomical Structures in XRA Closer Look at Glass BrainTVA Surg Medical Imaging VR ModuleData Visualization Is for Everyone: Open Source–Based Data Visualization in XRProtein Data VisualizationHands-On Tutorials: How to Create Data Visualization in Spatial ComputingHow to Create Data Visualization: ResourcesConclusionReferencesResourcesData Visualization ToolsMachine Learning Visualization ToolsData Journalism VisualizationsGlossary
IntroductionBehaviorsCurrent Practice: Reactive AIAdaptabilityComplexity and UniversalityFeasibilityMore Intelligence in the System: Deliberative AIMachine LearningReinforcement LearningDeep Reinforcement LearningImitation LearningCombining Automated Planning and Machine LearningApplicationsConclusionReferences
VR/AR Health Technology Application DesignStandard UX Isn’t IntuitivePick a Calm EnvironmentConvenienceTutorial: Insight Parkinson’s ExperimentWhat Insight DoesHow It Was BuiltCompaniesPlanning and GuidanceExperiences Designed for Medical EducationExperiences Designed for Use by PatientsProactive HealthCase Studies from Leading Academic Institutions
IntroductionPart 1: Five Key Principles of AR and VR for SportsNothing Is LivePart 2: The Next Evolution of Sports ExperiencesPart 3: Making the FutureOwnershipFinal ThoughtConclusion
Introduction: The Importance of Enterprise TrainingDoes VR Training Work?Use Case: Flood House TrainingWhat Is VR Training Good for? R.I.D.E.What Makes Good VR Training?Spherical VideoThe Benefits of Spherical VideoThe Challenges of Spherical VideoInteractions with Spherical VideoUse Case: Factory Floor TrainingThe Role of NarrativeUse Case: Store Robbery TrainingThe Future of XR Training: Beyond Spherical VideoComputer GraphicsUse Case: Soft Skills TrainingThe Future: PhotogrammetryThe Future: Light FieldsThe Future: AR TrainingThe Future: Voice RecognitionThe Future: The Ideal Training ScenarioReferences

Content preview from Creating Augmented and Virtual Realities

Chapter 7. Virtual Reality Toolkit: Open Source Framework for the Community

Harvey Ball and Clorama Dorvilias

Virtual Reality Toolkit (VRTK) is an open source, cross-platform toolkit for rapidly building virtual reality (VR) experiences by providing easy-to-use solutions to common problems. VRTK focuses on two main areas of help for developers: interactions and locomotion techniques, offering a multitude of ways of solving these common problems.

What Is VRTK and Why People Use It?

VRTK is an open source codebase that allows users to drag and drop functionality. By being able to drag and drop Assets into Uniity 3D, with a few configurations, they can immediately open example scenes with ready-set fully functional essential gameplay mechanics such as locomotion, navmeshes, a variety of user interfaces, and physical interactions around which you can start building your game.

With the benefits of it being open source, anyone can immediately reduce their setup time and immediately begin customizing assets and the source code to manifest their ideas in Unity for rapid prototyping—at the very least.

The major benefits to this toolkit are that it’s the only one of its kind that is readily adaptable to any hardware on which you plan to develop: Oculus + Touch controllers, HTC Vive, mixed reality (MR) headsets, mobile VR headsets. Because hardware accessibility can be a barrier to many aspiring or new VR creators, the default of its use includes a VR simulator. The VR simulator allows ...