Converting RDDs to DataFrames
In this section, we will discuss the strategies exposed by Spark SQL for transforming existing RDDs into DataFrames.
In today's enterprise world, data analysis requires the usage of more than one tool or technology. There could be scenarios where we want the Spark batch to initially load and process the data for a few insights and at the same we also want Spark SQL to process the same data to get the rest of the insights. In these kinds of scenarios, data would be loaded only once, either by a Spark batch or Spark SQL, and then it will be further processed by other Spark extensions. We need to consider that loading the data twice will be a waste of memory and time.
In order to solve this problem, Spark SQL (DataFrames) ...
Get Real-Time Big Data Analytics now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
