Deep Learning Cookbook

Preface

A Brief History of Deep Learning

The roots of the current deep learning boom go surprisingly far back, to the 1950s. While vague ideas of “intelligent machines” can be found further back in fiction and speculation, the 1950s and ’60s saw the introduction of the first “artificial neural networks,” based on a dramatically simplified model of biological neurons. Amongst these models, the Perceptron system articulated by Frank Rosenblatt garnered particular interest (and hype). Connected to a simple “camera” circuit, it could learn to distinguish different types of objects. Although the first version ran as software on an IBM computer, subsequent versions were done in pure hardware.

Interest in the multilayer perceptron (MLP) model continued through the ’60s. This changed when, in 1969, Marvin Minksy and Seymour Papert published their book Perceptrons (MIT Press). The book contained a proof showing that linear perceptrons could not classify the behavior of a nonlinear function (XOR). Despite the limitations of the proof (nonlinear perceptron models existed at the time of the book’s publication, and are even noted by the authors), its publication heralded the plummeting of funding for neural network models. Research would not recover until the 1980s, with the rise of a new generation of researchers.

The increase in computing power together with the development of the back-propagation technique (known in various forms since the ’60s, but not applied in general until the ’80s) ...