Apache Spark 2.x Cookbook

Getting ready

The power of nested structures goes far beyond traditional use cases, though. It has been very difficult to represent hierarchical data in highly normalized databases. Data needs to be joined across tables as needed. This does provide us with flexibility. Let's understand it with the example we covered in the previous recipe. In the Yelp dataset, a user reviews a business, which is represented by yelp_academic_dataset_review.json. In reality, a user reviews multiple businesses and a business is reviewed by multiple users. One would argue that it represents standard NxN relationships between entities, so what's the big deal here? The challenges come in how distributed systems operate. To make a join happen, the data needs to ...