PySpark Cookbook

How to do it...

There are two ways to produce histograms in PySpark:

• Select feature you want to visualize, .collect() it on the driver, and then use the matplotlib's native .hist(...) method to draw the histogram
• Calculate the counts in each histogram bin in PySpark and only return the counts to the driver for visualization

The former solution will work for small datasets (such as ours in this chapter) but it will break your driver if the data is too big. Moreover, there's a good reason why we distribute the data so we can do the computations in parallel instead of in a single thread. Thus, in this recipe, we will only show you the second solution. Here's the snippet that does all the calculations for us:

histogram_MPG = (    no_outliers