Data Science on the Google Cloud Platform, 2nd Edition

Chapter 12. The Full Dataset

In Chapters 1–11, we built a system for predicting flight delays so as to provide travelers with guidance on whether they would be likely to make it to their already scheduled meetings. All of the development was carried out on one year of data. In this chapter, I will change the code to process the full dataset.

All of the code snippets in this chapter are available in the folder 12_fulldataset of the book’s GitHub repository.

Four Years of Data

How well the final model performs can be evaluated only on truly independent data. Because we used our “test” data to evaluate different models along the way and do hyperparameter tuning, we cannot use any of the originally ingested data to evaluate the performance of the model.

Fortunately, though, I did not actually use all of the available data. In order to keep the datasets small enough that the Dataflow pipelines and ML training jobs would complete in a couple hours, I have limited all my work so far to 2015. I have not used 2016–2021 data in training, model selection, or hyperparameter tuning.

Let’s fix this. What I am going to do is to train the ML model on data from 2015–2018 and assume that we put the model into production at the end of 2018. How would that model have fared in 2019? If this works well, it gives us the confidence that we can train the ML model on a few years of data and then apply it in real time. That said, you probably realize why I’m not training the model on 2015–2020 and testing ...