Introducing .NET for Apache Spark: Distributed Processing for Massive Datasets

Appendix A: Running in the Cloud

When we have developed our .NET for Apache Spark application, we will likely want to run it on a production cluster, which means running in the cloud for many organizations today. When we want to run our application in the cloud, we have three options:

1. 1.

   Use Databricks, which is a fully managed Apache Spark instance by the creators of Apache Spark.

    
2. 2.

   Use a third-party implementation such as AWS EMR, Azure Synapse Analytics, or GCP Dataproc.

    
3. 3.

   Create and manage your clusters.

   In this appendix, we will take a look at what Databricks is and how we can use it to manage and execute our .NET for Apache Spark applications.

    

Databricks

Databricks is available in Amazon Web Services (AWS) and Azure but not available in ...