Microsoft HoloLens Developer’s Guide

Microsoft HoloLens Developer’s Guide

by Dennis Vroegop
Released July 2017
Publisher(s): Packt Publishing
ISBN: 9781786460851

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Transform the ways you communicate, create, collaborate, and explore using Microsoft HoloLens

About This Book

  • Create immersive augmented reality apps for Microsoft HoloLens from scratch

  • Leverage the powerful HoloLens sensors to interact with real-world motions and gestures and make your app life-like

  • Explore the powerful Unity 5 SDK along with the Windows Unified platform to get the most out of your HoloLens app

    • Who This Book Is For

    If you are a developer who wants to create augmented reality apps for the Microsoft HoloLens platform, then this is the book for you. Coding experience with C# is assumed.

    What You Will Learn

  • Design an app for HoloLens that is feasible and attractive to use

  • Add gestures and interact with them

  • Create sounds in the app and place them in a 3D space

  • Use voice generation and voice recognition to make your apps more lifelike

  • Interact with the physical environment to place holograms on top of physical objects

  • Compare HoloLens with the other products and know how to use its strengths

  • Use assets from third parties to enrich our app

    • In Detail

    HoloLens, Microsoft’s innovative augmented reality headset, overlaps holograms into a user’s vision of their environment. Your ideas are closer to becoming real when you can create and work with holograms in relation to the world around you. If you are dreaming beyond virtual worlds, beyond screens, beyond pixels, and want to take a big leap in the world of augmented reality, then this is the book you want.

    Starting off with brainstorming and the design process, you will take your first steps in creating your application for HoloLens. You will learn to add gestures and write an app that responds to verbal commands before gradually moving on creating sounds in the app and placing them in a 3D space. You will then communicate between devices in the boundaries of the UWP model.

    Style and approach

    This book takes a step-by-step, practical, tutorial-style approach where you will dive deep into HoloLens app development. You will work with the API and write your own complex scripts that would interact with the powerful HoloLens sensors and with realistic examples, you will be able to create immersive 3D apps for HoloLens.

    Table of contents

    1. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the example code
        2. Downloading the color images of this book
        3. Errata
        4. Piracy
        5. Questions
    2. Lets Begin!
      1. Welcome to the HoloLens!
        1. The device
          1. A wearable computer
          2. The sensors
            1. Cameras
            2. Microphones
            3. Holographic lenses
            4. Spatial speakers
          3. Processors
          4. The device in a nutshell
          5. Displaying 3D images
          6. Wearing the device
          7. Turning the device on for the first time
        2. Learning the gestures
        3. Calibrating the device
      2. The device portal
        1. Overview of the portal
        2. The different menus in the portal
          1. The Views menu in the portal
      3. Writing your first holographic application
        1. The tools we need to write software
        2. Your very first application
          1. Deploying your first app to the emulator
          2. How to use the emulator
          3. Deploying the app to the device
            1. Deploying through Wi-Fi
            2. Deploying through USB
          4. Let's have a look at the code
            1. The structure of the project
            2. Inside the AppView class
      4. Using Unity3D to create great holographic apps
        1. Our first HoloApp in Unity3D
          1. Fixing the camera
            1. Changing the position
            2. Changing the background color
          2. Adding holographic objects
          3. Create a sample script
          4. Building the project
          5. Continuing in Visual Studio after Unity
        2. The structure of a Unity HoloLens UWP app solution
          1. Build configurations with Unity
          2. Building and running your HoloLens Unity app
      5. Summary
    3. Designing Your App
      1. Unique features of the device
        1. Thinking in three dimensions
          1. Making sure objects are in a logical place
          2. Making sure objects can be reached
          3. Making sure objects can be found
          4. Pointers
          5. Sounds
        2. Dealing with limited input methods
          1. Gestures
          2. Audio
        3. People do not expect sudden changes
        4. The social aspect in collaborative situations
        5. Give your user superpowers!
      2. Forming a design team
        1. Big projects need big teams
        2. Small projects need small teams
      3. Using your experience
        1. Where holographic apps are "just another app"
        2. Where do holographic apps differ?
      4. The workflow to identify your features
        1. Writing down your mission statement
        2. Brainstorming all ideas
        3. Removing ideas that don't support the mission statement
        4. Separating must-haves from nice-to-haves
        5. Prioritizing must-haves
        6. Implementing your app
      5. Introducing the RockOn app
        1. What do we want to achieve?
        2. How do we get there?
          1. Decision time - tooling
            1. Developing with DirectX
            2. Developing with Unity3D
          2. Creating a new project
            1. Getting started with Unity3D
            2. The role of Visual Studio in developing for the HoloLens
          3. Adding assets to the scene
          4. Building the project
          5. Deploying the project
            1. Using the emulator
            2. Using the device
          6. Making some small adjustments
      6. Summary
    4. Gestures and Events
      1. Working with gestures
      2. Some UX considerations
        1. A brief history of user interfaces
          1. Command-line interface
          2. Graphical user interface globally
          3. Natural user interface
        2. Feedback
          1. Types of feedback in the HoloLens world
      3. Getting to grips with gaze
        1. Gaze in DirectX
          1. The bare HoloCursor class
          2. A word about shaders
            1. The processing pipeline
            2. Programming shaders
            3. Adding shaders to our project
            4. Additional data
          3. Defining images in three dimensions.
            1. Our square cursor
          4. Rendering and updating
            1. The Render() method
            2. Updating after changes
            3. Using our new class
            4. Positioning
            5. Rotating
          5. Next steps
        2. Gaze in Unity
          1. The cursor
            1. Creating the assets
            2. Editing the script in Visual Studio
            3. Using the script in Unity
            4. Implementing the cursor code
            5. Testing for touch
      4. Events
        1. Gestures in DirectX
          1. The InteractionManager class
          2. The SpatialGestureRecognizer
        2. Gestures in Unity
          1. Creating a prefab
            1. Creating the creation code
      5. Summary
    5. Make Some Noise - Sounds
      1. How our bodies work when it comes to sound
        1. Head Related Transfer Function
          1. Left and right
          2. Front and back
          3. Up and down
          4. Head Related Transfer Functions
          5. Moving our heads
          6. Echo
        2. Spatial sound in computer programs
      2. Sound design
        1. The design principles for spatial sound
          1. The benefits of spatial sound
            1. Grounding
            2. User attention
            3. Immersion
            4. Interaction design
          2. Best practices in sound design
        2. Applying these principles
          1. Licensing
      3. Background music
        1. DirectX
          1. Getting the right NuGet packages
          2. Loading and preparing the sound we want to play
          3. Playing the sound
          4. Actually play the sounds
        2. Unity
      4. Spatial sound
        1. DirectX
          1. The SpatialSound library
          2. Using the library
        2. Unity
          1. Setting the audio settings in Unity3D
          2. Adding spatial sound to GameObjects
          3. Play the sound at the right time.
      5. Some final thoughts on spatial sound
        1. General concepts when it comes to sounds
      6. What is next?
      7. Summary
    6. Be Heard - Voice Interactions
      1. Let the HoloLens speak
        1. General principles behind voice generation
        2. Voice generation in DirectX
          1. Creating the base of the VoicePlayer class
          2. Initializing the sound system for streaming
          3. Implementing IDispose the right way
          4. Playing the sound
          5. Synthesizing voice
          6. Setting things up
          7. Calling Play to hear the voice
          8. Making the voice spatial
        3. Voice generation in Unity3D
          1. Creating the Unity app
          2. Adding the helper script
          3. Creating the scripts for the asset
          4. Our SpatialVoicePlayer script
            1. What is it about these #if statements?
            2. Asynchronous stream generating
            3. Creating the reusable asset
          5. Adding voice to RockOn
        4. Making things more lifelike
          1. Introducing the SSML stream
          2. Overview of the most important SSML tags
      2. Listen up - voice commands
        1. Default commands
          1. Select
          2. Hey, Cortana
          3. App commands
          4. Hologram manipulations
          5. Dictation
        2. Troubleshooting speech recognition
        3. Building a VoiceRecognizer in DirectX
          1. The VoiceRecognizer class
        4. Building the VoiceRecognizer in Unity3D
        5. What is next?
      3. Summary
    7. Take a Look around You - Spatial Mapping
      1. VR, AR, and MR
        1. Virtual reality
        2. Augmented reality
        3. Mixed reality
        4. Usages of spatial mapping
          1. Occlusion
          2. Navigation
          3. Placement
          4. Visualization
          5. Physics
          6. Masquerading imperfection
        5. Things to be aware of in spatial mapping
          1. Errors in the spatial mapping
            1. Holes
            2. Fake surfaces
            3. Misaligned surfaces
          2. Causes for errors
            1. The user's movements
            2. The physical qualities of the surfaces
            3. Lighting
            4. Moving objects
          3. App-specific requirements
          4. Optimizing meshes
      2. Spatial understanding in DirectX
        1. Initial app
          1. Spatial capabilities
          2. Initial classes
          3. Checking capabilities
          4. Setting up device-dependent resources
          5. Creating the SpatialSurfaceObserver
          6. Getting the surface data
          7. Getting the actual mesh data
          8. Pruning the list
          9. The SurfaceMesh class
          10. Updating and rendering
          11. Mesh updating and rendering
          12. Cleaning up
          13. Putting it all to work
      3. Spatial understanding in Unity3D
        1. Writing scripts for spatial mapping in Unity3D
        2. Adding surface mapping
          1. Capabilities
          2. Spatial mapping in code
          3. Handling changes in the surfaces
          4. New and updated surfaces
          5. Wireframe
          6. The easy way
      4. Occlusion
      5. Summary
    8. Lets Talk! Communications between Devices
      1. The position of holograms
        1. Local coordinate systems
        2. Spatial mapping and coordinate systems
          1. Spatial anchors
            1. Using the spatial anchors in DirectX
            2. Using SpatialAnchors in Unity3D
            3. Changing GestureHandler.cs
        3. Problems with anchors
        4. Stationary frame of reference
          1. Using the stationary frame of reference in DirectX
          2. Using the stationary frame of reference in Unity3D
        5. Spatial locator-attached frame of reference
          1. Using the locator attached frame in DirectX
          2. What this does?
          3. Using the locator-attached frame in Unity3D
      2. Sharing
        1. Setting up networking
      3. Sharing scenarios
        1. Using a hub
          1. Setting up the data
          2. Setting up the TCP/IP layer
          3. Creating the thread for handling data traffic
          4. Handling connections
          5. Sending data
          6. Creating the mixed-reality client app
          7. Setting up the connection
          8. Consuming the connection
          9. Connecting from Unity3D
        2. Peer to peer
          1. Introducing Bluetooth advertisements
          2. Setting up the advertisement
          3. Watching for other devices
          4. Handling found devices
          5. Connecting to the other HoloLens
      4. Sharing world coordinates
        1. Sharing spatial anchors in DirectX
          1. Should you send one or multiple anchors?
          2. Sharing world anchors in Unity3D
            1. Sending the data from Unity3D
            2. Receiving the data in Unity3D
      5. Summary
    9. Speed Up Your Development - Advanced Unity3D
      1. The position of holograms
        1. Local coordinate systems
        2. Spatial mapping and coordinate systems
          1. Spatial anchors
            1. Using the spatial anchors in DirectX
            2. Using SpatialAnchors in Unity3D
            3. Changing GestureHandler.cs
        3. Problems with anchors
        4. Stationary frame of reference
          1. Using the stationary frame of reference in DirectX
          2. Using the stationary frame of reference in Unity3D
        5. Spatial locator-attached frame of reference
          1. Using the locator attached frame in DirectX
          2. What this does?
          3. Using the locator-attached frame in Unity3D
      2. Sharing
        1. Setting up networking
      3. Sharing scenarios
        1. Using a hub
          1. Setting up the data
          2. Setting up the TCP/IP layer
          3. Creating the thread for handling data traffic
          4. Handling connections
          5. Sending data
          6. Creating the mixed-reality client app
          7. Setting up the connection
          8. Consuming the connection
          9. Connecting from Unity3D
        2. Peer to peer
          1. Introducing Bluetooth advertisements
          2. Setting up the advertisement
          3. Watching for other devices
          4. Handling found devices
          5. Connecting to the other HoloLens
      4. Sharing world coordinates
        1. Sharing spatial anchors in DirectX
          1. Should you send one or multiple anchors?
          2. Sharing world anchors in Unity3D
            1. Sending the data from Unity3D
            2. Receiving the data in Unity3D
      5. Summary

    Product information

    • Title: Microsoft HoloLens Developer’s Guide
    • Author(s): Dennis Vroegop
    • Release date: July 2017
    • Publisher(s): Packt Publishing
    • ISBN: 9781786460851