book

Scala: Guide for Data Science Professionals

Name: Scala: Guide for Data Science Professionals
ISBN: 9781787282858

by Pascal Bugnion, Arun Manivannan, Patrick R. Nicolas

February 2017

Beginner to intermediate

1100 pages

25h 19m

English

Packt Publishing

Read now

Unlock full access

Scala: Guide for Data Science Professionals
Table of Contents
Scala: Guide for Data Science Professionals
Scala: Guide for Data Science Professionals
Credits
Preface
What this learning path covers
What you need for this learning path
Who this learning path is for
Reader feedback
Customer support
Downloading the example codeErrataPiracyQuestions

1. Module 1
1. Scala and Data Science
Data science
Programming in data science
Why Scala?
Static typing and type inferenceScala encourages immutabilityScala and functional programsNull pointer uncertaintyEasier parallelismInteroperability with Java
When not to use Scala
Summary
References
2. Manipulating Data with Breeze
Code examples
Installing Breeze
Getting help on Breeze
Basic Breeze data types
VectorsDense and sparse vectors and the vector traitMatricesBuilding vectors and matricesAdvanced indexing and slicingMutating vectors and matricesMatrix multiplication, transposition, and the orientation of vectorsData preprocessing and feature engineeringBreeze – function optimizationNumerical derivativesRegularization
An example – logistic regression
Towards re-usable code
Alternatives to Breeze
Summary
References
3. Plotting with breeze-viz
Diving into Breeze
Customizing plots
Customizing the line type
More advanced scatter plots
Multi-plot example – scatterplot matrix plots
Managing without documentation
Breeze-viz reference
Data visualization beyond breeze-viz
Summary
4. Parallel Collections and Futures
Parallel collectionsLimitations of parallel collectionsError handlingSetting the parallelism levelAn example – cross-validation with parallel collections
Futures
Future composition – using a future's resultBlocking until completionControlling parallel execution with execution contextsFutures example – stock price fetcher
Summary
References
5. Scala and SQL through JDBC
Interacting with JDBC
First steps with JDBC
Connecting to a database serverCreating tablesInserting dataReading data
JDBC summary
Functional wrappers for JDBC
Safer JDBC connections with the loan pattern
Enriching JDBC statements with the "pimp my library" pattern
Wrapping result sets in a stream
Looser coupling with type classes
Type classesCoding against type classesWhen to use type classesBenefits of type classes
Creating a data access layer
Summary
References
6. Slick – A Functional Interface for SQL
FEC dataImporting SlickDefining the schemaConnecting to the databaseCreating tablesInserting dataQuerying data
Invokers
Operations on columns
Aggregations with "Group by"
Accessing database metadata
Slick versus JDBC
Summary
References
7. Web APIs
A whirlwind tour of JSON
Querying web APIs
JSON in Scala – an exercise in pattern matching
JSON4S typesExtracting fields using XPath
Extraction using case classes
Concurrency and exception handling with futures
Authentication – adding HTTP headers
HTTP – a whirlwind overviewAdding headers to HTTP requests in Scala
Summary
References
8. Scala and MongoDB
MongoDB
Connecting to MongoDB with Casbah
Connecting with authentication
Inserting documents
Extracting objects from the database
Complex queries
Casbah query DSL
Custom type serialization
Beyond Casbah
Summary
References
9. Concurrency with Akka
GitHub follower graph
Actors as people
Hello world with Akka
Case classes as messages
Actor construction
Anatomy of an actor
Follower network crawler
Fetcher actors
Routing
Message passing between actors
Queue control and the pull pattern
Accessing the sender of a message
Stateful actors
Follower network crawler
Fault tolerance
Custom supervisor strategies
Life-cycle hooks
What we have not talked about
Summary
References
10. Distributed Batch Processing with Spark
Installing Spark
Acquiring the example data
Resilient distributed datasets
RDDs are immutableRDDs are lazyRDDs know their lineageRDDs are resilientRDDs are distributedTransformations and actions on RDDsPersisting RDDsKey-value RDDsDouble RDDs
Building and running standalone programs
Running Spark applications locallyReducing logging output and Spark configurationRunning Spark applications on EC2
Spam filtering
Lifting the hood
Data shuffling and partitions
Summary
Reference
11. Spark SQL and DataFrames
DataFrames – a whirlwind introduction
Aggregation operations
Joining DataFrames together
Custom functions on DataFrames
DataFrame immutability and persistence
SQL statements on DataFrames
Complex data types – arrays, maps, and structs
StructsArraysMaps
Interacting with data sources
JSON filesParquet files
Standalone programs
Summary
References
12. Distributed Machine Learning with MLlib
Introducing MLlib – Spam classification
Pipeline components
TransformersEstimators
Evaluation
Regularization in logistic regression
Cross-validation and model selection
Beyond logistic regression
Summary
References
13. Web APIs with Play
Client-server applications
Introduction to web frameworks
Model-View-Controller architecture
Single page applications
Building an application
The Play framework
Dynamic routing
Actions
Composing the responseUnderstanding and parsing the request
Interacting with JSON
Querying external APIs and consuming JSON
Calling external web servicesParsing JSONAsynchronous actions
Creating APIs with Play: a summary
Rest APIs: best practice
Summary
References
14. Visualization with D3 and the Play Framework
GitHub user data
Do I need a backend?
JavaScript dependencies through web-jars
Towards a web application: HTML templates
Modular JavaScript through RequireJS
Bootstrapping the applications
Client-side program architecture
Designing the modelThe event busAJAX calls through JQueryResponse views
Drawing plots with NVD3
Summary
References
A. Pattern Matching and Extractors
Pattern matching in for comprehensions
Pattern matching internals
Extracting sequences
Summary
Reference
II. Module 2
1. Getting Started with Breeze
Introduction
Getting Breeze – the linear algebra library
How to do it...There's more...The org.scalanlp.breeze dependencyThe org.scalanlp.breeze-natives package
Working with vectors
Getting readyHow to do it...Creating vectorsConstructing a vector from valuesCreating a zero vectorCreating a vector out of a functionCreating a vector of linearly spaced valuesCreating a vector with values in a specific rangeCreating an entire vector with a single valueSlicing a sub-vector from a bigger vectorCreating a Breeze Vector from a Scala VectorVector arithmeticScalar operationsCalculating the dot product of two vectorsCreating a new vector by adding two vectors togetherAppending vectors and converting a vector of one type to anotherConcatenating two vectorsConverting a vector of Int to a vector of DoubleComputing basic statisticsMean and varianceStandard deviationFind the largest value in a vectorFinding the sum, square root and log of all the values in the vectorThe Sqrt functionThe Log function
Working with matrices
How to do it...Creating matricesCreating a matrix from valuesCreating a zero matrixCreating a matrix out of a functionCreating an identity matrixCreating a matrix from random numbersCreating from a Scala collectionMatrix arithmeticAdditionMultiplicationAppending and conversionConcatenating matrices – verticallyConcatenating matrices – horizontallyConverting a matrix of Int to a matrix of DoubleData manipulation operationsGetting column vectors out of the matrixGetting row vectors out of the matrixGetting values inside the matrixGetting the inverse and transpose of a matrixComputing basic statisticsMean and varianceStandard deviationFinding the largest value in a matrixFinding the sum, square root and log of all the values in the matrixSqrtLogCalculating the eigenvectors and eigenvalues of a matrixHow it works...
Vectors and matrices with randomly distributed values
How it works...Creating vectors with uniformly distributed random valuesCreating vectors with normally distributed random valuesCreating vectors with random values that have a Poisson distributionCreating a matrix with uniformly random valuesCreating a matrix with normally distributed random valuesCreating a matrix with random values that has a Poisson distribution
Reading and writing CSV files
How it works...
2. Getting Started with Apache Spark DataFrames
Introduction
Getting Apache Spark
How to do it...
Creating a DataFrame from CSV
How to do it...How it works...There's more…
Manipulating DataFrames
How to do it...Printing the schema of the DataFrameSampling the data in the DataFrameSelecting DataFrame columnsFiltering data by conditionSorting data in the frameRenaming columnsTreating the DataFrame as a relational tableJoining two DataFramesInner joinRight outer joinLeft outer joinSaving the DataFrame as a file
Creating a DataFrame from Scala case classes
How to do it...How it works...
3. Loading and Preparing Data – DataFrame
Introduction
Loading more than 22 features into classes
How to do it...How it works...There's more…
Loading JSON into DataFrames
How to do it…Reading a JSON file using SQLContext.jsonFileReading a text file and converting it to JSON RDDExplicitly specifying your schemaThere's more…
Storing data as Parquet files
How to do it…Load a simple CSV file, convert it to case classes, and create a DataFrame from itSave it as a Parquet fileInstall Parquet toolsUsing the tools to inspect the Parquet fileEnable compression for the Parquet file
Using the Avro data model in Parquet
How to do it…Creation of the Avro modelGeneration of Avro objects using the sbt-avro pluginConstructing an RDD of our generated object from Students.csvSaving RDD[StudentAvro] in a Parquet fileReading the file back for verificationUsing Parquet tools for verification
Loading from RDBMS
How to do it…
Preparing data in Dataframes
How to do it...
4. Data Visualization
Introduction
Visualizing using Zeppelin
How to do it...Installing ZeppelinCustomizing Zeppelin's server and websocket portVisualizing data on HDFS – parameterizing inputsRunning custom functionsAdding external dependencies to ZeppelinPointing to an external Spark cluster
Creating scatter plots with Bokeh-Scala
How to do it...Preparing our dataCreating Plot and Document objectsCreating a marker objectSetting the X and Y axes' data range for the plotDrawing the x and the y axesViewing flower species with varying colorsAdding grid linesAdding a legend to the plot
Creating a time series MultiPlot with Bokeh-Scala
How to do it...Preparing our dataCreating a plotCreating a line that joins all the data pointsSetting the x and y axes' data range for the plotDrawing the axes and the gridsAdding toolsAdding a legend to the plotMultiple plots in the document
5. Learning from Data
Introduction
Supervised and unsupervised learning
Gradient descent
Predicting continuous values using linear regression
How to do it...Importing the dataConverting each instance into a LabeledPointPreparing the training and test dataScaling the featuresTraining the modelPredicting against test dataEvaluating the modelRegularizing the parametersMini batching
Binary classification using LogisticRegression and SVM
How to do it...Importing the dataTokenizing the data and converting it into LabeledPointsFactoring the inverse document frequencyPrepare the training and test dataConstructing the algorithmTraining the model and predicting the test dataEvaluating the model
Binary classification using LogisticRegression with Pipeline API
How to do it...Importing and splitting data as test and training setsConstruct the participants of the PipelinePreparing a pipeline and training a modelPredicting against test dataEvaluating a model without cross-validationConstructing parameters for cross-validationConstructing cross-validator and fit the best modelEvaluating the model with cross-validation
Clustering using K-means
How to do it...KMeans.RANDOMKMeans.PARALLELK-means++K-means||Max iterationsEpsilonImporting the data and converting it into a vectorFeature scaling the dataDeriving the number of clustersConstructing the modelEvaluating the model
Feature reduction using principal component analysis
How to do it...Dimensionality reduction of data for supervised learningMean-normalizing the training dataExtracting the principal componentsPreparing the labeled dataPreparing the test dataClassify and evaluate the metricsDimensionality reduction of data for unsupervised learningMean-normalizing the training dataExtracting the principal componentsArriving at the number of componentsEvaluating the metrics
6. Scaling Up
Introduction
Building the Uber JAR
How to do it...Transitive dependency stated explicitly in the SBT dependencyTwo different libraries depend on the same external library
Submitting jobs to the Spark cluster (local)
How to do it...Downloading SparkRunning HDFS on Pseudo-clustered modeRunning the Spark master and slave locallyPushing data into HDFSSubmitting the Spark application on the cluster
Running the Spark Standalone cluster on EC2
How to do it...Creating the AccessKey and pem fileSetting the environment variablesRunning the launch scriptVerifying installationMaking changes to the codeTransferring the data and job filesLoading the dataset into HDFSRunning the jobDestroying the cluster
Running the Spark Job on Mesos (local)
How to do it...Installing MesosStarting the Mesos master and slaveUploading the Spark binary package and the dataset to HDFSRunning the job
Running the Spark Job on YARN (local)
How to do it...Installing the Hadoop clusterStarting HDFS and YARNPushing Spark assembly and dataset to HDFSRunning a Spark job in yarn-client modeRunning Spark job in yarn-cluster mode
7. Going Further
Introduction
Using Spark Streaming to subscribe to a Twitter stream
How to do it...
Using Spark as an ETL tool
How to do it...
Using StreamingLogisticRegression to classify a Twitter stream using Kafka as a training stream
How to do it...
Using GraphX to analyze Twitter data
How to do it...
III. Module 3
1. Getting Started
Mathematical notation for the curious
Why machine learning?
ClassificationPredictionOptimizationRegression
Why Scala?
AbstractionScalabilityConfigurabilityMaintainabilityComputation on demand
Model categorization
Taxonomy of machine learning algorithms
Unsupervised learningClusteringDimension reductionSupervised learningGenerative modelsDiscriminative modelsReinforcement learning
Tools and frameworks
JavaScalaApache Commons MathDescriptionLicensingInstallationJFreeChartDescriptionLicensingInstallationOther libraries and frameworks
Source code
Context versus view boundsPresentationPrimitives and implicitsPrimitive typesType conversionsOperatorsImmutabilityPerformance of Scala iterators
Let's kick the tires
Overview of computational workflowsWriting a simple workflowSelecting a datasetLoading the datasetPreprocessing the datasetBasic statisticsNormalization and Gauss distributionPlotting dataCreating a model (learning)Classify the data
Summary
2. Hello World!
ModelingA model by any other nameModel versus designSelecting a model's featuresExtracting features
Designing a workflow
The computational frameworkThe pipe operatorMonadic data transformationDependency injectionWorkflow modulesThe workflow factoryExamples of workflow componentsThe preprocessing moduleThe clustering module
Assessing a model
ValidationKey metricsImplementationK-fold cross-validationBias-variance decompositionOverfitting
Summary
3. Data Preprocessing
Time series
Moving averages
The simple moving averageThe weighted moving averageThe exponential moving average
Fourier analysis
Discrete Fourier transform (DFT)DFT-based filteringDetection of market cycles
The Kalman filter
The state space estimationThe transition equationThe measurement equationThe recursive algorithmPredictionCorrectionKalman smoothingExperimentation
Alternative preprocessing techniques
Summary
4. Unsupervised Learning
ClusteringK-means clusteringMeasuring similarityOverview of the K-means algorithmStep 1 – cluster configurationDefining clustersDefining K-meansInitializing clustersStep 2 – cluster assignmentStep 3 – iterative reconstructionCurse of dimensionalityExperimentTuning the number of clustersValidationExpectation-maximization (EM) algorithmGaussian mixture modelEM overviewImplementationTestingOnline EM
Dimension reduction
Principal components analysis (PCA)AlgorithmImplementationTest caseEvaluationOther dimension reduction techniques
Performance considerations
K-meansEMPCA
Summary
5. Naïve Bayes Classifiers
Probabilistic graphical models
Naïve Bayes classifiers
Introducing the multinomial Naïve BayesFormalismThe frequentist perspectiveThe predictive modelThe zero-frequency problemImplementationSoftware designTrainingClassificationLabelingResults
Multivariate Bernoulli classification
ModelImplementation
Naïve Bayes and text mining
Basics of information retrievalImplementationExtraction of termsScoring of termsTestingRetrieving textual informationEvaluation
Pros and cons
Summary
6. Regression and Regularization
Linear regressionOne-variate linear regressionImplementationTest caseOrdinary least squares (OLS) regressionDesignImplementationTest case 1 – trendingTest case 2 – features selection
Regularization
Ln roughness penaltyThe ridge regressionImplementationThe test case
Numerical optimization
The logistic regression
The logit functionBinomial classificationSoftware designThe training workflowConfiguring the least squares optimizerComputing the Jacobian matrixDefining the exit conditionsDefining the least squares problemMinimizing the loss functionTestClassification
Summary
7. Sequential Data Models
Markov decision processesThe Markov propertyThe first-order discrete Markov chain
The hidden Markov model (HMM)
NotationThe lambda modelHMM execution stateEvaluation (CF-1)Alpha class (the forward variable)Beta class (the backward variable)Training (CF-2)Baum-Welch estimator (EM)Decoding (CF-3)The Viterbi algorithmPutting it all togetherTest caseThe hidden Markov model for time series analysis
Conditional random fields
Introduction to CRFLinear chain CRF
CRF and text analytics
The feature functions modelSoftware designImplementationBuilding the training setGenerating tagsExtracting data sequencesCRF control parametersPutting it all togetherTestsThe training convergence profileImpact of the size of the training setImpact of the L2 regularization factor
Comparing CRF and HMM
Performance consideration
Summary
8. Kernel Models and Support Vector Machines
Kernel functionsOverviewCommon discriminative kernels
The support vector machine (SVM)
The linear SVMThe separable case (hard margin)The nonseparable case (soft margin)The nonlinear SVMMax-margin classificationThe kernel trick
Support vector classifier (SVC)
The binary SVCLIBSVMSoftware designConfiguration parametersSVM FormulationThe SVM kernel functionSVM executionSVM implementationC-penalty and marginKernel evaluationApplication to risk analysisFeatures and labels
Anomaly detection with one-class SVC
Support vector regression (SVR)
OverviewSVR versus linear regression
Performance considerations
Summary
9. Artificial Neural Networks
Feed-forward neural networks (FFNN)The Biological backgroundThe mathematical background
The multilayer perceptron (MLP)
The activation functionThe network architectureSoftware designModel definitionLayersSynapsesConnectionsTraining cycle/epochStep 1 – input forward propagationThe computational modelObjectiveSoftmaxStep 2 – sum of squared errorsStep 3 – error backpropagationError propagationThe computational modelStep 4 – synapse/weights adjustmentMomentum factor for gradient descentImplementationStep 5 – convergence criteriaConfigurationPutting all togetherTraining strategies and classificationOnline versus batch trainingRegularizationModel instantiationPrediction
Evaluation
Impact of learning rateImpact of the momentum factorTest caseImplementationModels evaluationImpact of hidden layers architecture
Benefits and limitations
Summary
10. Genetic Algorithms
EvolutionThe originNP problemsEvolutionary computing
Genetic algorithms and machine learning
Genetic algorithm components
EncodingsValue encodingPredicate encodingSolution encodingThe encoding schemeFlat encodingHierarchical encodingGenetic operatorsSelectionCrossoverMutationFitness score
Implementation
Software designKey componentsSelectionControlling population growthGA configurationCrossoverPopulationChromosomesGenesMutationPopulationChromosomesGenesThe reproduction cycle
GA for trading strategies
Definition of trading strategiesTrading operatorsThe cost/unfitness functionTrading signalsTrading strategiesSignal encodingTest caseData extractionInitial populationConfigurationGA instantiationGA executionTestsThe unweighted scoreThe weighted score
Advantages and risks of genetic algorithms
Summary
11. Reinforcement Learning
IntroductionThe problemA solution – Q-learningTerminologyConceptValue of policyBellman optimality equationsTemporal difference for model-free learningAction-value iterative updateImplementationSoftware designStates and actionsSearch spacePolicy and action-valueThe Q-learning trainingTail recursion to the rescuePredictionOption trading using Q-learningOption propertyOption modelFunction approximationConstrained state-transitionPutting it all togetherEvaluationPros and cons of reinforcement learning
Learning classifier systems
Introduction to LCSWhy LCSTerminologyExtended learning classifier systems (XCS)XCS componentsApplication to portfolio managementXCS core dataXCS rulesCoveringExample of implementationBenefits and limitation of learning classifier systems
Summary
12. Scalable Frameworks
Overview
Scala
Controlling object creationParallel collectionsProcessing a parallel collectionBenchmark frameworkPerformance evaluation
Scalability with Actors
The Actor modelPartitioningBeyond actors – reactive programming
Akka
Master-workersMessages exchangeWorker actorsThe workflow controllerThe master ActorMaster with routingDistributed discrete Fourier transformLimitationsFuturesThe Actor life cycleBlocking on futuresHandling future callbacksPutting all together
Apache Spark
Why SparkDesign principlesIn-memory persistencyLazinessTransforms and ActionsShared variablesExperimenting with SparkDeploying SparkUsing Spark shellMLlibRDD generationK-means using SparkPerformance evaluationTuning parametersTestsPerformance considerationsPros and cons0xdata Sparkling Water
Summary
B. Basic Concepts
Scala programmingList of librariesFormat of code snippetsEncapsulationClass constructor templateCompanion objects versus case classesEnumerations versus case classesOverloadingDesign template for classifiersData extractionData sourcesExtraction of documentsMatrix class
Mathematics
Linear algebraQR DecompositionLU factorizationLDL decompositionCholesky factorizationSingular value decompositionEigenvalue decompositionAlgebraic and numerical librariesFirst order predicate logicJacobian and Hessian matricesSummary of optimization techniquesGradient descent methodsSteepest descentConjugate gradientStochastic gradient descentQuasi-Newton algorithmsBFGSL-BFGSNonlinear least squares minimizationGauss-NewtonLevenberg-MarquardtLagrange multipliersOverview of dynamic programming
Finances 101
Fundamental analysisTechnical analysisTerminologyTrading signals and strategyPrice patternsOptions tradingFinancial data sources
Suggested online courses
References
C. Bibliography
Index

Content preview from Scala: Guide for Data Science Professionals

Apache Spark

Apache Spark is a fast and general-purpose cluster computing system, initially developed as AMPLab / UC Berkley as part of the Berkeley Data Analytics Stack (BDAS), http://en.wikipedia.org/wiki/UC_Berkeley. It provides high-level APIs for the following programming languages that make large, concurrent parallel jobs easy to write and deploy [12:11]:

Scala: http://spark.apache.org/docs/latest/api/scala/index.html
Java: http://spark.apache.org/docs/latest/api/java/index.html
Python: http://spark.apache.org/docs/latest/api/python/index.html

Tip

Link to latest information

The URLs as any reference to Apache Spark may change in future versions.

The core element of Spark is Resilient Distributed Dataset (RDD), which is a collection of elements ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781787282858Purchase Link

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Scala: Guide for Data Science Professionals

by Pascal Bugnion, Arun Manivannan, Patrick R. Nicolas

Apache Spark

Tip

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

More than 5,000 organizations count on O’Reilly

Julian F.

Addison B.

Amir M.

Mark W.

You might also like

Scala for Data Science

Learning Scala

Scala Reactive Programming

Scala Data Analysis Cookbook (new)

Publisher Resources