Practical Simulations for Machine Learning

Chapter 10. Using Cameras in Simulations

It’s time to get some real vision. And we don’t mean the shiny robot man, we mean cameras (and lights, and action). In this chapter, we’re going to look at how you can use a camera that sees the world of your simulation as an observation for your agents.

No longer will your observations be bound by feeding your agent numbers from code, and from sensors! Instead, you’ll be bound by what the camera you choose to set up can see. (Which, if we’re getting technical, are also numbers, but we digress.)

Observations and Camera Sensors

So far, all the observations we’ve been using are basically numbers—usually Vector3s—that we’ve been providing to the agent via our CollectObservations method, or collecting by using sensors of some kind, looking into our environment to measure things, or using grid-based observations for a 2D spatial representation of things.

We’ve either implemented CollectObservations() in our Agent, and passed in vectors, and other forms of numbers, or we’ve added components to the agent in the Unity Editor, which have—under the hood—created raycasts (perfect lasers to measure distances and what they hit) and automatically passed the numbers arising from those into the ML-Agents system.

There’s another way to provide observations to our agent: by using CameraSensor and RenderTextureSensor. These allow us to pass image information, in the form of a 3D Tensor, to the convolutional neural network (CNN) of the agent policy. So, basically, ...