Data Science Solutions on Azure: Tools and Techniques Using Databricks and MLOps

© Julian Soh and Priyanshi Singh 2020

J. Soh, P. SinghData Science Solutions on Azurehttps://doi.org/10.1007/978-1-4842-6405-8_6

6. Apache Spark, Big Data, and Azure Databricks

Julian Soh¹   and Priyanshi Singh²

(1)

Olympia, WA, USA

(2)

New Jersey, NJ, USA

 

The exponential pace of innovation in artificial intelligence in recent years can be attributed to advancements in machine learning. In turn, the advancements in machine learning are based on two core developments – availability of data and ubiquitous access to unparalleled compute capabilities.

In Chapter 2 and elsewhere in this book, you have seen the recurring theme that the more data that is available to train models, the better the models are at predicting outcomes and help us make better decisions. ...