Data Science and Engineering at Enterprise Scale

Chapter 6. Advanced Machine Learning Examples and Applications

In this chapter we will provide advanced machine learning use cases, along with examples of how to use them. The examples here are slightly outside of the canon of Apache Spark machine learning, but are extremely helpful if you have occasion to use them.

Deep Learning Models with Spark and TensorFlow

The field of deep learning is yet another topic that is so vast that only a small portion of it can be covered here. In principle, deep learning is simply another type of supervised learning, as it almost always requires as its inputs a set of labeled data and features. There is a training stage and a prediction phase, and the workflow described earlier in Figure 5-3 is just as relevant to deep learning models as any supervised learning protocols.

The deep learning models themselves are, however, much more complex, and the software and hardware required for these models are typically outside the Spark ecosystem. There are many other very good frameworks to choose from, but we will provide an example with the very popular TensorFlow framework. The complexity of the inputs can also increase vastly, as well as the amount of data used to train the models. The inputs can have thousands of features, with millions of parameters, and the size of the training sets can be orders of magnitude larger than the classic machine learning workflows. With models this large, it is difficult to interpret the parameters in any intuitive way, ...