Scala: Guide for Data Science Professionals

Summary

In this chapter, we established the framework for the different data processing units that will be introduced in this book. There is a very good reason why the topics of model validation and overfitting are explored early on in this book. There is no point in building models and selecting algorithms if we do not have a methodology to evaluate their relative merits.

In this chapter, you were introduced to:

• The versatility and cleanness of the Cake pattern in Scala as an effective scaffolding tool for data processing
• The concept of pipe operator for data conversion
• A robust methodology to validate machine learning models
• The challenge in fitting models to both training and real-world data

The next chapter will address the problem of overfitting ...