book

Big Data Analytics

Name: Big Data Analytics
ISBN: 9781785884696

by Venkat Ankam, Aravind Nallan

September 2016

Beginner to intermediate

326 pages

6h 47m

English

Packt Publishing

Read now

Unlock full access

Big Data Analytics
Table of Contents
Big Data Analytics
Credits
About the Author
Acknowledgement
About the Reviewers
www.PacktPub.com
eBooks, discount offers, and moreWhy subscribe?
Preface
What this book covers
What you need for this book

Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example codeDownloading the color images of this bookErrataPiracyQuestions
1. Big Data Analytics at a 10,000-Foot View
Big Data analytics and the role of Hadoop and SparkA typical Big Data analytics project life cycleIdentifying the problem and outcomesIdentifying the necessary dataData collectionPreprocessing data and ETLPerforming analyticsVisualizing dataThe role of Hadoop and Spark
Big Data science and the role of Hadoop and Spark
A fundamental shift from data analytics to data scienceData scientists versus software engineersData scientists versus data analystsData scientists versus business analystsA typical data science project life cycleHypothesis and modelingMeasuring the effectivenessMaking improvementsCommunicating the resultsThe role of Hadoop and Spark
Tools and techniques
Real-life use cases
Summary
2. Getting Started with Apache Hadoop and Apache Spark
Introducing Apache HadoopHadoop Distributed File SystemFeatures of HDFSMapReduceMapReduce featuresMapReduce v1 versus MapReduce v2MapReduce v1 challengesYARNStorage options on HadoopFile formatsSequence fileProtocol buffers and thriftAvroParquetRCFile and ORCFileCompression formatsStandard compression formats
Introducing Apache Spark
Spark historyWhat is Apache Spark?What Apache Spark is notMapReduce issuesSpark's stack
Why Hadoop plus Spark?
Hadoop featuresSpark featuresFrequently asked questions about Spark
Installing Hadoop plus Spark clusters
Summary
3. Deep Dive into Apache Spark
Starting Spark daemonsWorking with CDHWorking with HDP, MapR, and Spark pre-built packages
Learning Spark core concepts
Ways to work with SparkSpark ShellExploring the Spark Scala shellSpark applicationsConnecting to the Kerberos Security Enabled Spark ClusterResilient Distributed DatasetMethod 1 – parallelizing a collectionMethod 2 – reading from a fileReading files from HDFSReading files from HDFS with HA enabledSpark contextTransformations and actionsParallelism in RDDsLazy evaluationLineage GraphSerializationLeveraging Hadoop file formats in SparkData localityShared variablesPair RDDs
Lifecycle of Spark program
PipeliningSpark execution summary
Spark applications
Spark Shell versus Spark applicationsCreating a Spark contextSparkConfSparkSubmitSpark Conf precedence orderImportant application configurations
Persistence and caching
Storage levelsWhat level to choose?
Spark resource managers – Standalone, YARN, and Mesos
Local versus cluster modeCluster resource managersStandaloneYARNDynamic resource allocationClient mode versus cluster modeMesosWhich resource manager to use?
Summary
4. Big Data Analytics with Spark SQL, DataFrames, and Datasets
History of Spark SQL
Architecture of Spark SQL
Introducing SQL, Datasources, DataFrame, and Dataset APIs
Evolution of DataFrames and Datasets
What's wrong with RDDs?RDD Transformations versus Dataset and DataFrames Transformations
Why Datasets and DataFrames?
OptimizationSpeedAutomatic Schema DiscoveryMultiple sources, multiple languagesInteroperability between RDDs and othersSelect and read necessary data only
When to use RDDs, Datasets, and DataFrames?
Analytics with DataFrames
Creating SparkSessionCreating DataFramesCreating DataFrames from structured data filesCreating DataFrames from RDDsCreating DataFrames from tables in HiveCreating DataFrames from external databasesConverting DataFrames to RDDsCommon Dataset/DataFrame operationsInput and Output OperationsBasic Dataset/DataFrame functionsDSL functionsBuilt-in functions, aggregate functions, and window functionsActionsRDD operationsCaching dataPerformance optimizations
Analytics with the Dataset API
Creating DatasetsConverting a DataFrame to a DatasetConverting a Dataset to a DataFrameAccessing metadata using Catalog
Data Sources API
Read and write functionsBuilt-in sourcesWorking with text filesWorking with JSONWorking with ParquetWorking with ORCWorking with JDBCWorking with CSVExternal sourcesWorking with AVROWorking with XMLWorking with PandasDataFrame based Spark-on-HBase connector
Spark SQL as a distributed SQL engine
Spark SQL's Thrift server for JDBC/ODBC accessQuerying data using beeline clientQuerying data from Hive using spark-sql CLIIntegration with BI tools
Hive on Spark
Summary
5. Real-Time Analytics with Spark Streaming and Structured Streaming
Introducing real-time processingPros and cons of Spark StreamingHistory of Spark Streaming
Architecture of Spark Streaming
Spark Streaming application flowStateless and stateful stream processing
Spark Streaming transformations and actions
UnionJoinTransform operationupdateStateByKeymapWithStateWindow operationsOutput operations
Input sources and output stores
Basic sourcesAdvanced sourcesCustom sourcesReceiver reliabilityOutput stores
Spark Streaming with Kafka and HBase
Receiver-based approachRole of ZookeeperDirect approach (no receivers)Integration with HBase
Advanced concepts of Spark Streaming
Using DataFramesMLlib operationsCaching/persistenceFault-tolerance in Spark StreamingFailure of executorFailure of driverRecovering with checkpointingRecovering with WALPerformance tuning of Spark Streaming applications
Monitoring applications
Introducing Structured Streaming
Structured Streaming application flowWhen to use Structured Streaming?Streaming Datasets and Streaming DataFramesInput sources and output sinksOperations on Streaming Datasets and Streaming DataFrames
Summary
6. Notebooks and Dataflows with Spark and Hadoop
Introducing web-based notebooks
Introducing Jupyter
Installing JupyterAnalytics with Jupyter
Introducing Apache Zeppelin
Jupyter versus ZeppelinInstalling Apache ZeppelinAmbari serviceThe manual methodAnalytics with Zeppelin
The Livy REST job server and Hue Notebooks
Installing and configuring the Livy server and HueUsing the Livy serverAn interactive sessionA batch sessionSharing SparkContexts and RDDsUsing Livy with Hue NotebookUsing Livy with Zeppelin
Introducing Apache NiFi for dataflows
Installing Apache NiFiDataflows and analytics with NiFi
Summary
7. Machine Learning with Spark and Hadoop
Introducing machine learning
Machine learning on Spark and Hadoop
Machine learning algorithms
Supervised learningUnsupervised learningRecommender systemsFeature extraction and transformationOptimizationSpark MLlib data types
An example of machine learning algorithms
Logistic regression for spam detection
Building machine learning pipelines
An example of a pipeline workflowBuilding an ML pipelineSaving and loading models
Machine learning with H2O and Spark
Why Sparkling Water?An application flow on YARNGetting started with Sparkling Water
Introducing Hivemall
Introducing Hivemall for Spark
Summary
8. Building Recommendation Systems with Spark and Mahout
Building recommendation systemsContent-based filteringCollaborative filteringUser-based collaborative filteringItem-based collaborative filtering
Limitations of a recommendation system
A recommendation system with MLlib
Preparing the environmentCreating RDDsExploring the data with DataFramesCreating training and testing datasetsCreating a modelMaking predictionsEvaluating the model with testing dataChecking the accuracy of the modelExplicit versus implicit feedback
The Mahout and Spark integration
Installing MahoutExploring the Mahout shellBuilding a universal recommendation system with Mahout and search tool
Summary
9. Graph Analytics with GraphX
Introducing graph processingWhat is a graph?Graph databases versus graph processing systemsIntroducing GraphXGraph algorithms
Getting started with GraphX
Basic operations of GraphXCreating a graphCountingFilteringinDegrees, outDegrees, and degreesTripletsTransforming graphsTransforming attributesModifying graphsJoining graphsVertexRDD and EdgeRDD operationsMapping VertexRDD and EdgeRDDFiltering VertexRDDsJoining VertexRDDsJoining EdgeRDDsReversing edge directionsGraphX algorithmsTriangle countingConnected components
Analyzing flight data using GraphX
Pregel API
Introducing GraphFrames
Motif findingLoading and saving GraphFrames
Summary
10. Interactive Analytics with SparkR
Introducing R and SparkRWhat is R?Introducing SparkRArchitecture of SparkR
Getting started with SparkR
Installing and configuring RUsing SparkR shellLocal modeStandalone modeYarn modeCreating a local DataFrameCreating a DataFrame from a DataSources APICreating a DataFrame from HiveUsing SparkR scripts
Using DataFrames with SparkR
Using SparkR with RStudio
Machine learning with SparkR
Using the Naive Bayes modelUsing the k-means model
Using SparkR with Zeppelin
Summary
Index

Content preview from Big Data Analytics

Learning Spark core concepts

Let's understand the core concepts of Spark in this section. The main abstraction Spark provides is a Resilient Distributed Dataset (RDD). So, let's understand what an RDD is and operations in RDDs that provide in-memory performance and fault tolerance. But, let's learn the ways to work with Spark first.

Ways to work with Spark

There are a couple of ways to work with Spark—Spark Shell and Spark Applications.

Spark Shell

Interactive REPL (read-eval-print loop) for data exploration using Scala, Python, or R:

// Entering to Scala Shell . :q to exit the shell.
[cloudera@quickstart spark-2.0.0-bin-hadoop2.7]$ bin/spark-shell

# Entering to Python Shell. ctrl+d to exit the shell. 
[cloudera@quickstart spark-2.0.0-bin-hadoop2.7]$ ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781785884696

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Big Data Analytics

by Venkat Ankam, Aravind Nallan

Learning Spark core concepts