Machine Learning with Spark - Second Edition

Creating a streaming data producer

The data producer operates in a manner similar to our product event producer example. Recall from Chapter 5, Building a Recommendation Engine with Spark, that a linear model is a linear combination (or vector dot product) of a weight vector, w, and a feature vector, x (that is, wTx). Our producer will generate synthetic data using a fixed, known weight vector and randomly generated feature vectors. This data fits the linear model formulation exactly, so we will expect our regression model to learn the true weight vector fairly easily.

First, we will set up a maximum number of events per second (say, 100) and the number of features in our feature vector (also 100 in this example):

/**  * A producer application ...