book

Scala Data Analysis Cookbook (new)

Name: Scala Data Analysis Cookbook (new)
Author: Arun Manivannan
ISBN: 9781784396749

by Arun Manivannan

October 2015

Beginner to intermediate

254 pages

5h 31m

English

Packt Publishing

Read now

Unlock full access

Scala Data Analysis Cookbook
Table of Contents
Scala Data Analysis Cookbook
Credits
About the Author
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers, and moreWhy Subscribe?Free Access for Packt account holders
Preface
Apache Flink
Scalding
Saddle

Spire
Akka
Accord
What this book covers
What you need for this book
Who this book is for
Sections
Getting readyHow to do it…How it works…There's more…See also
Conventions
Reader feedback
Customer support
Downloading the example codeErrataPiracyQuestions
1. Getting Started with Breeze
Introduction
Getting Breeze – the linear algebra library
How to do it...There's more...The org.scalanlp.breeze dependencyThe org.scalanlp.breeze-natives package
Working with vectors
Getting readyHow to do it...Creating vectorsConstructing a vector from valuesCreating a zero vectorCreating a vector out of a functionCreating a vector of linearly spaced valuesCreating a vector with values in a specific rangeCreating an entire vector with a single valueSlicing a sub-vector from a bigger vectorCreating a Breeze Vector from a Scala VectorVector arithmeticScalar operationsCalculating the dot product of two vectorsCreating a new vector by adding two vectors togetherAppending vectors and converting a vector of one type to anotherConcatenating two vectorsConverting a vector of Int to a vector of DoubleComputing basic statisticsMean and varianceStandard deviationFind the largest value in a vectorFinding the sum, square root and log of all the values in the vectorThe Sqrt functionThe Log function
Working with matrices
How to do it...Creating matricesCreating a matrix from valuesCreating a zero matrixCreating a matrix out of a functionCreating an identity matrixCreating a matrix from random numbersCreating from a Scala collectionMatrix arithmeticAdditionMultiplicationAppending and conversionConcatenating matrices – verticallyConcatenating matrices – horizontallyConverting a matrix of Int to a matrix of DoubleData manipulation operationsGetting column vectors out of the matrixGetting row vectors out of the matrixGetting values inside the matrixGetting the inverse and transpose of a matrixComputing basic statisticsMean and varianceStandard deviationFinding the largest value in a matrixFinding the sum, square root and log of all the values in the matrixSqrtLogCalculating the eigenvectors and eigenvalues of a matrixHow it works...
Vectors and matrices with randomly distributed values
How it works...Creating vectors with uniformly distributed random valuesCreating vectors with normally distributed random valuesCreating vectors with random values that have a Poisson distributionCreating a matrix with uniformly random valuesCreating a matrix with normally distributed random valuesCreating a matrix with random values that has a Poisson distribution
Reading and writing CSV files
How it works...
2. Getting Started with Apache Spark DataFrames
Introduction
Getting Apache Spark
How to do it...
Creating a DataFrame from CSV
How to do it...How it works...There's more…
Manipulating DataFrames
How to do it...Printing the schema of the DataFrameSampling the data in the DataFrameSelecting DataFrame columnsFiltering data by conditionSorting data in the frameRenaming columnsTreating the DataFrame as a relational tableJoining two DataFramesInner joinRight outer joinLeft outer joinSaving the DataFrame as a file
Creating a DataFrame from Scala case classes
How to do it...How it works...
3. Loading and Preparing Data – DataFrame
Introduction
Loading more than 22 features into classes
How to do it...How it works...There's more…
Loading JSON into DataFrames
How to do it…Reading a JSON file using SQLContext.jsonFileReading a text file and converting it to JSON RDDExplicitly specifying your schemaThere's more…
Storing data as Parquet files
How to do it…Load a simple CSV file, convert it to case classes, and create a DataFrame from itSave it as a Parquet fileInstall Parquet toolsUsing the tools to inspect the Parquet fileEnable compression for the Parquet file
Using the Avro data model in Parquet
How to do it…Creation of the Avro modelGeneration of Avro objects using the sbt-avro pluginConstructing an RDD of our generated object from Students.csvSaving RDD[StudentAvro] in a Parquet fileReading the file back for verificationUsing Parquet tools for verification
Loading from RDBMS
How to do it…
Preparing data in Dataframes
How to do it...
4. Data Visualization
Introduction
Visualizing using Zeppelin
How to do it...Installing ZeppelinCustomizing Zeppelin's server and websocket portVisualizing data on HDFS – parameterizing inputsRunning custom functionsAdding external dependencies to ZeppelinPointing to an external Spark cluster
Creating scatter plots with Bokeh-Scala
How to do it...Preparing our dataCreating Plot and Document objectsCreating a marker objectSetting the X and Y axes' data range for the plotDrawing the x and the y axesViewing flower species with varying colorsAdding grid linesAdding a legend to the plot
Creating a time series MultiPlot with Bokeh-Scala
How to do it...Preparing our dataCreating a plotCreating a line that joins all the data pointsSetting the x and y axes' data range for the plotDrawing the axes and the gridsAdding toolsAdding a legend to the plotMultiple plots in the document
5. Learning from Data
Introduction
Supervised and unsupervised learning
Gradient descent
Predicting continuous values using linear regression
How to do it...Importing the dataConverting each instance into a LabeledPointPreparing the training and test dataScaling the featuresTraining the modelPredicting against test dataEvaluating the modelRegularizing the parametersMini batching
Binary classification using LogisticRegression and SVM
How to do it...Importing the dataTokenizing the data and converting it into LabeledPointsFactoring the inverse document frequencyPrepare the training and test dataConstructing the algorithmTraining the model and predicting the test dataEvaluating the model
Binary classification using LogisticRegression with Pipeline API
How to do it...Importing and splitting data as test and training setsConstruct the participants of the PipelinePreparing a pipeline and training a modelPredicting against test dataEvaluating a model without cross-validationConstructing parameters for cross-validationConstructing cross-validator and fit the best modelEvaluating the model with cross-validation
Clustering using K-means
How to do it...KMeans.RANDOMKMeans.PARALLELK-means++K-means||Max iterationsEpsilonImporting the data and converting it into a vectorFeature scaling the dataDeriving the number of clustersConstructing the modelEvaluating the model
Feature reduction using principal component analysis
How to do it...Dimensionality reduction of data for supervised learningMean-normalizing the training dataExtracting the principal componentsPreparing the labeled dataPreparing the test dataClassify and evaluate the metricsDimensionality reduction of data for unsupervised learningMean-normalizing the training dataExtracting the principal componentsArriving at the number of componentsEvaluating the metrics
6. Scaling Up
Introduction
Building the Uber JAR
How to do it...Transitive dependency stated explicitly in the SBT dependencyTwo different libraries depend on the same external library
Submitting jobs to the Spark cluster (local)
How to do it...Downloading SparkRunning HDFS on Pseudo-clustered modeRunning the Spark master and slave locallyPushing data into HDFSSubmitting the Spark application on the cluster
Running the Spark Standalone cluster on EC2
How to do it...Creating the AccessKey and pem fileSetting the environment variablesRunning the launch scriptVerifying installationMaking changes to the codeTransferring the data and job filesLoading the dataset into HDFSRunning the jobDestroying the cluster
Running the Spark Job on Mesos (local)
How to do it...Installing MesosStarting the Mesos master and slaveUploading the Spark binary package and the dataset to HDFSRunning the job
Running the Spark Job on YARN (local)
How to do it...Installing the Hadoop clusterStarting HDFS and YARNPushing Spark assembly and dataset to HDFSRunning a Spark job in yarn-client modeRunning Spark job in yarn-cluster mode
7. Going Further
Introduction
Using Spark Streaming to subscribe to a Twitter stream
How to do it...
Using Spark as an ETL tool
How to do it...
Using StreamingLogisticRegression to classify a Twitter stream using Kafka as a training stream
How to do it...
Using GraphX to analyze Twitter data
How to do it...
Index

Content preview from Scala Data Analysis Cookbook (new)

Creating a DataFrame from CSV

In this recipe, we'll look at how to create a new DataFrame from a delimiter-separated values file.

Note

The code for this recipe can be found at https://github.com/arunma/ScalaDataAnalysisCookbook/blob/master/chapter1-spark-csv/src/main/scala/com/packt/scaladata/spark/csv/DataFrameCSV.scala.

How to do it...

This recipe involves four steps:

Add the spark-csv support to our project.
Create a Spark Config object that gives information on the environment that we are running Spark in.
Create a Spark context that serves as an entry point into Spark. Then, we proceed to create an SQLContext from the Spark context.
Load the CSV using the SQLContext.
CSV support isn't first-class in Spark, but it is available through an external library ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781784396749

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design