Data Science on the Google Cloud Platform

Chapter 9. Machine Learning Classifier Using TensorFlow

In Chapter 7, we built a machine learning model but ran into problems when trying to scale it out and make it operational. The first problem was how to prevent training–serving skew when using a time-windowed aggregate feature. We solved this in Chapter 8 by using the same code for computing the aggregates on historical data as will be used on real-time data. The Cloud Dataflow pipeline that we implemented in Chapter 8 was used to create two sets of files: trainFlights*.csv, which will serve as our training dataset for machine learning, and testFlights*.csv, which we will use to evaluate the model. Both of these files contain augmented datasets—the purpose of the pipeline was to add the computed time-aggregates to the raw data received from the airlines. We want to predict the first column in those files (whether the flight is on time) based on the other columns (departure delay, taxi-out time, distance, and average departure and arrival delays, and a few other fields).

While we solved the problem of dataset augmentation with time aggregates, the other three problems identified at the end of Chapter 7 remain:

• One-hot encoding categorical columns caused an explosion in the size of the dataset

• Embeddings would involve special bookkeeping

• Putting the model into production requires the machine learning library to be portable to environments beyond the cluster on which the model is trained.

The solution to these three ...