Data Science on the Google Cloud Platform, 2nd Edition

Conclusion

In Chapter 1, we discussed the goals of data analysis, how to provide data-driven guidance using statistical and machine learning models, and the roles that will be involved with such work in the future. We also formulated our case study problem—of recommending whether a traveler should cancel a scheduled meeting based on the likelihood of the flight that they are on being delayed.

In Chapter 2, we automated the ingest of flight data from the Bureau of Transportation Statistics website. We started out by reverse engineering a web form, writing Python scripts to download the necessary data, and storing the data on Google Cloud Storage. Finally, we made the ingest process serverless by creating a Cloud Run application to carry out the ingest and made it invokable from Cloud Scheduler.

In Chapter 3, we discussed why it was important to bring end users’ insights into our data modeling efforts as early as possible. We achieved this by building a dashboard in Data Studio and populated this dashboard from Cloud SQL. The dashboard was used to explain a simple contingency table model that predicted on-time arrival likelihood by thresholding the departure delay of the flight.

In Chapter 4, we simulated the flight data as if it were arriving in real time, used the simulation to populate messages into Cloud Pub/Sub, and then processed the streaming messages in Cloud Dataflow. In Cloud Dataflow, we computed aggregations and streamed the results into BigQuery. Because Cloud Dataflow ...