First, we created a dataset from a Scala list and displayed the output to validate the creation of the dataset as expected. Second, we loaded a comma-separated value (CSV) file into memory, transforming it into a dataset of type Team. Third, we executed the map() function over our dataset to build a list of team city names and printed out the execution plan used to generate the dataset. Finally, we persisted the teams dataset we previously loaded into a JSON formatted file for future use.
How it works...
Get Apache Spark 2.x Machine Learning Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
