book

Big Data Analytics

by Venkat Ankam, Aravind Nallan

September 2016

Beginner to intermediate

326 pages

6h 47m

English

Packt Publishing

Read now

Unlock full access

eBooks, discount offers, and moreWhy subscribe?
What this book covers

Downloading the example codeDownloading the color images of this bookErrataPiracyQuestions
Big Data analytics and the role of Hadoop and SparkA typical Big Data analytics project life cycleIdentifying the problem and outcomesIdentifying the necessary dataData collectionPreprocessing data and ETLPerforming analyticsVisualizing dataThe role of Hadoop and Spark
A fundamental shift from data analytics to data scienceData scientists versus software engineersData scientists versus data analystsData scientists versus business analystsA typical data science project life cycleHypothesis and modelingMeasuring the effectivenessMaking improvementsCommunicating the resultsThe role of Hadoop and Spark
Introducing Apache HadoopHadoop Distributed File SystemFeatures of HDFSMapReduceMapReduce featuresMapReduce v1 versus MapReduce v2MapReduce v1 challengesYARNStorage options on HadoopFile formatsSequence fileProtocol buffers and thriftAvroParquetRCFile and ORCFileCompression formatsStandard compression formats
Spark historyWhat is Apache Spark?What Apache Spark is notMapReduce issuesSpark's stack
Hadoop featuresSpark featuresFrequently asked questions about Spark
Starting Spark daemonsWorking with CDHWorking with HDP, MapR, and Spark pre-built packages
Ways to work with SparkSpark ShellExploring the Spark Scala shellSpark applicationsConnecting to the Kerberos Security Enabled Spark ClusterResilient Distributed DatasetMethod 1 – parallelizing a collectionMethod 2 – reading from a fileReading files from HDFSReading files from HDFS with HA enabledSpark contextTransformations and actionsParallelism in RDDsLazy evaluationLineage GraphSerializationLeveraging Hadoop file formats in SparkData localityShared variablesPair RDDs
PipeliningSpark execution summary
Spark Shell versus Spark applicationsCreating a Spark contextSparkConfSparkSubmitSpark Conf precedence orderImportant application configurations
Storage levelsWhat level to choose?
Local versus cluster modeCluster resource managersStandaloneYARNDynamic resource allocationClient mode versus cluster modeMesosWhich resource manager to use?
History of Spark SQL
What's wrong with RDDs?RDD Transformations versus Dataset and DataFrames Transformations
OptimizationSpeedAutomatic Schema DiscoveryMultiple sources, multiple languagesInteroperability between RDDs and othersSelect and read necessary data only
Creating SparkSessionCreating DataFramesCreating DataFrames from structured data filesCreating DataFrames from RDDsCreating DataFrames from tables in HiveCreating DataFrames from external databasesConverting DataFrames to RDDsCommon Dataset/DataFrame operationsInput and Output OperationsBasic Dataset/DataFrame functionsDSL functionsBuilt-in functions, aggregate functions, and window functionsActionsRDD operationsCaching dataPerformance optimizations
Creating DatasetsConverting a DataFrame to a DatasetConverting a Dataset to a DataFrameAccessing metadata using Catalog
Read and write functionsBuilt-in sourcesWorking with text filesWorking with JSONWorking with ParquetWorking with ORCWorking with JDBCWorking with CSVExternal sourcesWorking with AVROWorking with XMLWorking with PandasDataFrame based Spark-on-HBase connector
Spark SQL's Thrift server for JDBC/ODBC accessQuerying data using beeline clientQuerying data from Hive using spark-sql CLIIntegration with BI tools
Introducing real-time processingPros and cons of Spark StreamingHistory of Spark Streaming
Spark Streaming application flowStateless and stateful stream processing
UnionJoinTransform operationupdateStateByKeymapWithStateWindow operationsOutput operations
Basic sourcesAdvanced sourcesCustom sourcesReceiver reliabilityOutput stores
Receiver-based approachRole of ZookeeperDirect approach (no receivers)Integration with HBase
Using DataFramesMLlib operationsCaching/persistenceFault-tolerance in Spark StreamingFailure of executorFailure of driverRecovering with checkpointingRecovering with WALPerformance tuning of Spark Streaming applications
Structured Streaming application flowWhen to use Structured Streaming?Streaming Datasets and Streaming DataFramesInput sources and output sinksOperations on Streaming Datasets and Streaming DataFrames
Introducing web-based notebooks
Installing JupyterAnalytics with Jupyter
Jupyter versus ZeppelinInstalling Apache ZeppelinAmbari serviceThe manual methodAnalytics with Zeppelin
Installing and configuring the Livy server and HueUsing the Livy serverAn interactive sessionA batch sessionSharing SparkContexts and RDDsUsing Livy with Hue NotebookUsing Livy with Zeppelin
Installing Apache NiFiDataflows and analytics with NiFi
Introducing machine learning
Supervised learningUnsupervised learningRecommender systemsFeature extraction and transformationOptimizationSpark MLlib data types
Logistic regression for spam detection
An example of a pipeline workflowBuilding an ML pipelineSaving and loading models
Why Sparkling Water?An application flow on YARNGetting started with Sparkling Water
Building recommendation systemsContent-based filteringCollaborative filteringUser-based collaborative filteringItem-based collaborative filtering
Preparing the environmentCreating RDDsExploring the data with DataFramesCreating training and testing datasetsCreating a modelMaking predictionsEvaluating the model with testing dataChecking the accuracy of the modelExplicit versus implicit feedback
Installing MahoutExploring the Mahout shellBuilding a universal recommendation system with Mahout and search tool
Introducing graph processingWhat is a graph?Graph databases versus graph processing systemsIntroducing GraphXGraph algorithms
Basic operations of GraphXCreating a graphCountingFilteringinDegrees, outDegrees, and degreesTripletsTransforming graphsTransforming attributesModifying graphsJoining graphsVertexRDD and EdgeRDD operationsMapping VertexRDD and EdgeRDDFiltering VertexRDDsJoining VertexRDDsJoining EdgeRDDsReversing edge directionsGraphX algorithmsTriangle countingConnected components
Pregel API
Motif findingLoading and saving GraphFrames
Introducing R and SparkRWhat is R?Introducing SparkRArchitecture of SparkR
Installing and configuring RUsing SparkR shellLocal modeStandalone modeYarn modeCreating a local DataFrameCreating a DataFrame from a DataSources APICreating a DataFrame from HiveUsing SparkR scripts
Using the Naive Bayes modelUsing the k-means model

Overview

Dive into the world of big data with "Big Data Analytics: Real Time Analytics Using Apache Spark and Hadoop." This comprehensive guide introduces readers to the fundamentals and practical applications of Apache Spark and Hadoop, covering essential topics like Spark SQL, DataFrames, structured streaming, and more. Learn how to harness the power of real-time analytics and big data tools effectively.

What this Book will help me do

Master the key components of Apache Spark and Hadoop ecosystems, including Spark SQL and MapReduce.
Gain an understanding of DataFrames, DataSets, and structured streaming for seamless data handling.
Develop skills in real-time analytics using Spark Streaming and technologies like Kafka and HBase.
Learn to implement machine learning models using Spark's MLlib and ML Pipelines.
Explore graph analytics with GraphX and leverage data visualization tools like Jupyter and Zeppelin.

Author(s)

Venkat Ankam, an expert in big data technologies, has years of experience working with Apache Hadoop and Spark. As an educator and technical consultant, Venkat has enabled numerous professionals to gain critical insights into big data ecosystems. With a pragmatic approach, his writings aim to guide readers through complex systems in a structured and easy-to-follow manner.

Who is it for?

This book is perfect for data analysts, data scientists, software architects, and programmers aiming to expand their knowledge of big data analytics. Readers should ideally have a basic programming background in languages like Python, Scala, R, or SQL. Prior hands-on experience with big data environments is not necessary but is an added advantage. This guide is created to cater to a range of skill levels, from beginners to intermediate learners.