book

Data Analytics with Hadoop

by Benjamin Bengfort, Jenny Kim

June 2016

Intermediate to advanced

286 pages

8h 9m

English

O'Reilly Media, Inc.

Read now

Unlock full access

What to Expect from This BookWho This Book Is ForHow to Read This BookOverview of ChaptersProgramming and Code ExamplesGitHub RepositoryExecuting Distributed JobsPermissions and CitationFeedback and How to Contact UsSafari® Books OnlineHow to Contact UsAcknowledgments
What Is a Data Product?Building Data Products at Scale with HadoopLeveraging Large DatasetsHadoop for Data ProductsThe Data Science Pipeline and the Hadoop EcosystemBig Data WorkflowsConclusion
Basic ConceptsHadoop ArchitectureA Hadoop ClusterHDFSYARNWorking with a Distributed File SystemBasic File System OperationsFile Permissions in HDFSOther HDFS InterfacesWorking with Distributed ComputationMapReduce: A Functional Programming ModelMapReduce: Implemented on a ClusterBeyond a Map and Reduce: Job ChainingSubmitting a MapReduce Job to YARNConclusion
Hadoop StreamingComputing on CSV Data with StreamingExecuting Streaming JobsA Framework for MapReduce with PythonCounting BigramsOther FrameworksAdvanced MapReduceCombinersPartitionersJob ChainingConclusion
Spark BasicsThe Spark StackResilient Distributed DatasetsProgramming with RDDsInteractive Spark Using PySparkWriting Spark ApplicationsVisualizing Airline Delays with SparkConclusion
Computing with KeysCompound KeysKeyspace PatternsPairs versus StripesDesign PatternsSummarizationIndexingFilteringToward Last-Mile AnalyticsFitting a ModelValidating ModelsConclusion
Structured Data Queries with HiveThe Hive Command-Line Interface (CLI)Hive Query Language (HQL)Data Analysis with HiveHBaseNoSQL and Column-Oriented DatabasesReal-Time Analytics with HBaseConclusion
Importing Relational Data with SqoopImporting from MySQL to HDFSImporting from MySQL to HiveImporting from MySQL to HBaseIngesting Streaming Data with FlumeFlume Data FlowsIngesting Product Impression Data with FlumeConclusion

PigPig LatinData TypesRelational OperatorsUser-Defined FunctionsWrapping UpSpark’s Higher-Level APIsSpark SQLDataFramesConclusion
Scalable Machine Learning with SparkCollaborative FilteringClassificationClusteringConclusion
Data Product LifecycleData LakesData IngestionComputational Data StoresMachine Learning LifecycleConclusion
Quick StartSetting Up LinuxCreating a Hadoop UserConfiguring SSHInstalling JavaDisabling IPv6Installing HadoopUnpackingEnvironmentHadoop ConfigurationFormatting the NamenodeStarting HadoopRestarting Hadoop
Packaged Hadoop DistributionsSelf-Installation of Apache Hadoop Ecosystem ProductsBasic Installation and Configuration StepsSqoop-Specific ConfigurationsHive-Specific ConfigurationHBase-Specific ConfigurationsInstalling Spark

Content preview from Data Analytics with Hadoop

Part II. Workflows and Tools for Big Data Science

The second part of Data Analytics with Hadoop explores higher-level tools and workflows for practicing data scientists. Although a foundational knowledge of Hadoop, MapReduce, and Spark is required to understand what kind of analyses can be conducted at scale, the day-to-day efforts of a data scientist engaging in big data will generally revolve around the ecosystem of tools built on top of Hadoop. Generally speaking, we have organized these final chapters around the data product pipeline presented in Chapter 1.

Chapter 6 discusses data warehousing and data mining and introduces Hive and HBase for both relational and columnar data storage and queries. Chapter 7 identifies the need for ingestion utilities to get data into HDFS and looks at structured methods using Sqoop, as well as less structured ingestion using Flume. Chapter 8 explores higher-level APIs for analytics: Apache Pig and Spark DataFrames. Chapter 9 discusses machine learning and computational methods using Spark MLlib. Finally, Chapter 10 wraps things up and takes a complete view of doing data science by summarizing the integration of the workflows discussed in the previous chapters of this part.