Frank Kane's Taming Big Data with Apache Spark and Python

Getting results

Open up your shell, wherever it may be, and type in spark-submit spark-linear-regression.py to kick that off:

Here are our results:

You can see that our model is pretty good, there's a pretty good correlation between our actual observed values and the predictions, they tend to go in the right direction and they're not too far apart. There you have it, we basically constructed a linear model using MLlib, using DataFrames. The point here isn't really so much to show you how linear regression works, it's to show you how to use ...