Deep Learning Cookbook

Chapter 12. Image Style

In this chapter we’ll explore some techniques to visualize what convolutional networks see when they classify images. We’ll do this by running the networks in reverse—rather than giving the network an image and asking it what it is, we tell the network what to see and ask it to modify the image in a way that makes it see the detected item more exaggeratedly.

We’ll start by doing this for a single neuron. This will show us what sorts of patterns that neuron reacts to. We’ll then expand on this by introducing the concept of octaves, where we zoom in while we optimize the image to get more detail. Finally, we will look at applying this technique to existing images and visualize what the network “almost” sees in an image, a technique known as deep dreaming.

We’ll then switch gears and look at how combinations of “lower” layers of a network determine the artistic style of an image and how we can visualize just the style of an image. This uses the concept of gram matrices and how they represent the style of a painting.

Next, we look at how we can combine this notion with a way to stabilize an image, which allows us to generate an image that only copies the style of an image. We then move on to apply this technique to existing images, which makes it possible to render a recent photograph in the style of Vincent van Gogh’s Starry Skies. Finally, we’ll use two style images and render versions of the same picture somewhere between the two styles.

The following notebooks ...