book

Hadoop: The Definitive Guide, 4th Edition

by Tom White

April 2015

Beginner to intermediate

754 pages

21h 39m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Administrative NotesWhat’s New in the Fourth Edition?What’s New in the Third Edition?What’s New in the Second Edition?Conventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
Data!Data Storage and AnalysisQuerying All Your DataBeyond BatchComparison with Other SystemsRelational Database Management SystemsGrid ComputingVolunteer ComputingA Brief History of Apache HadoopWhat’s in This Book?
A Weather DatasetData FormatAnalyzing the Data with Unix ToolsAnalyzing the Data with HadoopMap and ReduceJava MapReduceA test runScaling OutData FlowCombiner FunctionsSpecifying a combiner functionRunning a Distributed MapReduce JobHadoop StreamingRubyPython
The Design of HDFSHDFS ConceptsBlocksNamenodes and DatanodesBlock CachingHDFS FederationHDFS High AvailabilityFailover and fencingThe Command-Line InterfaceBasic Filesystem OperationsHadoop FilesystemsInterfacesHTTPCNFSFUSEThe Java InterfaceReading Data from a Hadoop URLReading Data Using the FileSystem APIFSDataInputStreamWriting DataFSDataOutputStreamDirectoriesQuerying the FilesystemFile metadata: FileStatusListing filesFile patternsPathFilterDeleting DataData FlowAnatomy of a File ReadAnatomy of a File WriteCoherency ModelConsequences for application designParallel Copying with distcpKeeping an HDFS Cluster Balanced
Anatomy of a YARN Application RunResource RequestsApplication LifespanBuilding YARN ApplicationsYARN Compared to MapReduce 1Scheduling in YARNScheduler OptionsCapacity Scheduler ConfigurationQueue placementFair Scheduler ConfigurationEnabling the Fair SchedulerQueue configurationQueue placementPreemptionDelay SchedulingDominant Resource FairnessFurther Reading
Data IntegrityData Integrity in HDFSLocalFileSystemChecksumFileSystemCompressionCodecsCompressing and decompressing streams with CompressionCodecInferring CompressionCodecs using CompressionCodecFactoryNative librariesCodecPoolCompression and Input SplitsUsing Compression in MapReduceCompressing map outputSerializationThe Writable InterfaceWritableComparable and comparatorsWritable ClassesWritable wrappers for Java primitivesTextIndexingUnicodeIterationMutabilityResorting to StringBytesWritableNullWritableObjectWritable and GenericWritableWritable collectionsImplementing a Custom WritableImplementing a RawComparator for speedCustom comparatorsSerialization FrameworksSerialization IDLFile-Based Data StructuresSequenceFileWriting a SequenceFileReading a SequenceFileDisplaying a SequenceFile with the command-line interfaceSorting and merging SequenceFilesThe SequenceFile formatMapFileMapFile variantsOther File Formats and Column-Oriented Formats

The Configuration APICombining ResourcesVariable ExpansionSetting Up the Development EnvironmentManaging ConfigurationGenericOptionsParser, Tool, and ToolRunnerWriting a Unit Test with MRUnitMapperReducerRunning Locally on Test DataRunning a Job in a Local Job RunnerTesting the DriverRunning on a ClusterPackaging a JobThe client classpathThe task classpathPackaging dependenciesTask classpath precedenceLaunching a JobThe MapReduce Web UIThe resource manager pageThe MapReduce job pageRetrieving the ResultsDebugging a JobThe tasks and task attempts pagesHandling malformed dataHadoop LogsRemote DebuggingTuning a JobProfiling TasksThe HPROF profilerMapReduce WorkflowsDecomposing a Problem into MapReduce JobsJobControlApache OozieDefining an Oozie workflowPackaging and deploying an Oozie workflow applicationRunning an Oozie workflow job
Anatomy of a MapReduce Job RunJob SubmissionJob InitializationTask AssignmentTask ExecutionStreamingProgress and Status UpdatesJob CompletionFailuresTask FailureApplication Master FailureNode Manager FailureResource Manager FailureShuffle and SortThe Map SideThe Reduce SideConfiguration TuningTask ExecutionThe Task Execution EnvironmentStreaming environment variablesSpeculative ExecutionOutput CommittersTask side-effect files
MapReduce TypesThe Default MapReduce JobThe default Streaming jobKeys and values in StreamingInput FormatsInput Splits and RecordsFileInputFormatFileInputFormat input pathsFileInputFormat input splitsSmall files and CombineFileInputFormatPreventing splittingFile information in the mapperProcessing a whole file as a recordText InputTextInputFormatControlling the maximum line lengthKeyValueTextInputFormatNLineInputFormatXMLBinary InputSequenceFileInputFormatSequenceFileAsTextInputFormatSequenceFileAsBinaryInputFormatFixedLengthInputFormatMultiple InputsDatabase Input (and Output)Output FormatsText OutputBinary OutputSequenceFileOutputFormatSequenceFileAsBinaryOutputFormatMapFileOutputFormatMultiple OutputsAn example: Partitioning dataMultipleOutputsLazy OutputDatabase Output
CountersBuilt-in CountersTask countersJob countersUser-Defined Java CountersDynamic countersRetrieving countersUser-Defined Streaming CountersSortingPreparationPartial SortTotal SortSecondary SortJava codeStreamingJoinsMap-Side JoinsReduce-Side JoinsSide Data DistributionUsing the Job ConfigurationDistributed CacheUsageHow it worksThe distributed cache APIMapReduce Library Classes
Cluster SpecificationCluster SizingMaster node scenariosNetwork TopologyRack awarenessCluster Setup and InstallationInstalling JavaCreating Unix User AccountsInstalling HadoopConfiguring SSHConfiguring HadoopFormatting the HDFS FilesystemStarting and Stopping the DaemonsCreating User DirectoriesHadoop ConfigurationConfiguration ManagementEnvironment SettingsJavaMemory heap sizeSystem logfilesSSH settingsImportant Hadoop Daemon PropertiesHDFSYARNMemory settings in YARN and MapReduceCPU settings in YARN and MapReduceHadoop Daemon Addresses and PortsOther Hadoop PropertiesCluster membershipBuffer sizeHDFS block sizeReserved storage spaceTrashJob schedulerReduce slow startShort-circuit local readsSecurityKerberos and HadoopAn exampleDelegation TokensOther Security EnhancementsBenchmarking a Hadoop ClusterHadoop BenchmarksBenchmarking MapReduce with TeraSortOther benchmarksUser Jobs
HDFSPersistent Data StructuresNamenode directory structureThe filesystem image and edit logSecondary namenode directory structureDatanode directory structureSafe ModeEntering and leaving safe modeAudit LoggingToolsdfsadminFilesystem check (fsck)Finding the blocks for a fileDatanode block scannerBalancerMonitoringLoggingSetting log levelsGetting stack tracesMetrics and JMXMaintenanceRoutine Administration ProceduresMetadata backupsData backupsFilesystem check (fsck)Filesystem balancerCommissioning and Decommissioning NodesCommissioning new nodesDecommissioning old nodesUpgradesHDFS data and metadata upgradesStart the upgradeWait until the upgrade is completeCheck the upgradeRoll back the upgrade (optional)Finalize the upgrade (optional)
Avro Data Types and SchemasIn-Memory Serialization and DeserializationThe Specific APIAvro DatafilesInteroperabilityPython APIAvro ToolsSchema ResolutionSort OrderAvro MapReduceSorting Using Avro MapReduceAvro in Other Languages
Data ModelNested EncodingParquet File FormatParquet ConfigurationWriting and Reading Parquet FilesAvro, Protocol Buffers, and ThriftProjection and read schemasParquet MapReduce
Installing FlumeAn ExampleTransactions and ReliabilityBatchingThe HDFS SinkPartitioning and InterceptorsFile FormatsFan OutDelivery GuaranteesReplicating and Multiplexing SelectorsDistribution: Agent TiersDelivery GuaranteesSink GroupsIntegrating Flume with ApplicationsComponent CatalogFurther Reading
Getting SqoopSqoop ConnectorsA Sample ImportText and Binary File FormatsGenerated CodeAdditional Serialization SystemsImports: A Deeper LookControlling the ImportImports and ConsistencyIncremental ImportsDirect-Mode ImportsWorking with Imported DataImported Data and HiveImporting Large ObjectsPerforming an ExportExports: A Deeper LookExports and TransactionalityExports and SequenceFilesFurther Reading
Installing and Running PigExecution TypesLocal modeMapReduce modeRunning Pig ProgramsGruntPig Latin EditorsAn ExampleGenerating ExamplesComparison with DatabasesPig LatinStructureStatementsExpressionsTypesSchemasUsing Hive tables with HCatalogValidation and nullsSchema mergingFunctionsOther librariesMacrosUser-Defined FunctionsA Filter UDFLeveraging typesAn Eval UDFDynamic invokersA Load UDFUsing a schemaData Processing OperatorsLoading and Storing DataFiltering DataFOREACH...GENERATESTREAMGrouping and Joining DataJOINCOGROUPCROSSGROUPSorting DataCombining and Splitting DataPig in PracticeParallelismAnonymous RelationsParameter SubstitutionDynamic parametersParameter substitution processingFurther Reading
Installing HiveThe Hive ShellAn ExampleRunning HiveConfiguring HiveExecution enginesLoggingHive ServicesHive clientsThe MetastoreComparison with Traditional DatabasesSchema on Read Versus Schema on WriteUpdates, Transactions, and IndexesSQL-on-Hadoop AlternativesHiveQLData TypesPrimitive typesComplex typesOperators and FunctionsConversionsTablesManaged Tables and External TablesPartitions and BucketsPartitionsBucketsStorage FormatsThe default storage format: Delimited textBinary storage formats: Sequence files, Avro datafiles, Parquet files, RCFiles, and ORCFilesUsing a custom SerDe: RegexSerDeStorage handlersImporting DataInsertsMultitable insertCREATE TABLE...AS SELECTAltering TablesDropping TablesQuerying DataSorting and AggregatingMapReduce ScriptsJoinsInner joinsOuter joinsSemi joinsMap joinsSubqueriesViewsUser-Defined FunctionsWriting a UDFWriting a UDAFA more complex UDAFFurther Reading
An ExampleThe Core Crunch APIPrimitive Operationsunion()parallelDo()groupByKey()combineValues()TypesRecords and tuplesSources and TargetsReading from a sourceWriting to a targetExisting outputsCombined sources and targetsFunctionsSerialization of functionsObject reuseMaterializationPObjectPipeline ExecutionRunning a PipelineAsynchronous executionDebuggingStopping a PipelineInspecting a Crunch PlanIterative AlgorithmsCheckpointing a PipelineCrunch LibrariesFurther Reading
Installing SparkAn ExampleSpark Applications, Jobs, Stages, and TasksA Scala Standalone ApplicationA Java ExampleA Python ExampleResilient Distributed DatasetsCreationTransformations and ActionsAggregation transformationsPersistencePersistence levelsSerializationDataFunctionsShared VariablesBroadcast VariablesAccumulatorsAnatomy of a Spark Job RunJob SubmissionDAG ConstructionTask SchedulingTask ExecutionExecutors and Cluster ManagersSpark on YARNYARN client modeYARN cluster modeFurther Reading
HBasicsBackdropConceptsWhirlwind Tour of the Data ModelRegionsLockingImplementationHBase in operationInstallationTest DriveClientsJavaMapReduceREST and ThriftBuilding an Online Query ApplicationSchema DesignLoading DataLoad distributionBulk loadOnline QueriesStation queriesObservation queriesHBase Versus RDBMSSuccessful ServiceHBasePraxisHDFSUIMetricsCountersFurther Reading
Installing and Running ZooKeeperAn ExampleGroup Membership in ZooKeeperCreating the GroupJoining a GroupListing Members in a GroupZooKeeper command-line toolsDeleting a GroupThe ZooKeeper ServiceData ModelEphemeral znodesSequence numbersWatchesOperationsMultiupdateAPIsWatch triggersACLsImplementationConsistencySessionsTimeStatesBuilding Applications with ZooKeeperA Configuration ServiceThe Resilient ZooKeeper ApplicationInterruptedExceptionKeeperExceptionState exceptionsRecoverable exceptionsUnrecoverable exceptionsA reliable configuration serviceA Lock ServiceThe herd effectRecoverable exceptionsUnrecoverable exceptionsImplementationMore Distributed Data Structures and ProtocolsBookKeeper and HedwigZooKeeper in ProductionResilience and PerformanceConfigurationFurther Reading
From CPUs to Semantic IntegrationEnter Apache CrunchBuilding a Complete PictureIntegrating Healthcare DataComposability over FrameworksMoving Forward
The Structure of DNAThe Genetic Code: Turning DNA Letters into ProteinsThinking of DNA as Source CodeThe Human Genome Project and Reference GenomesSequencing and Aligning DNAADAM, A Scalable Genome Analysis PlatformLiterate programming with the Avro interface description language (IDL)Column-oriented access with ParquetA simple example: k-mer counting using Spark and ADAMFrom Personalized Ads to Personalized MedicineJoin In
Fields, Tuples, and PipesOperationsTaps, Schemes, and FlowsCascading in PracticeFlexibilityHadoop and Cascading at ShareThisSummary
PrerequisitesInstallationConfigurationStandalone ModePseudodistributed ModeConfiguring SSHFormatting the HDFS filesystemStarting and stopping the daemonsCreating a user directoryFully Distributed Mode

Content preview from Hadoop: The Definitive Guide, 4th Edition

Foreword

Doug Cutting, April 2009

Shed in the Yard, California

Hadoop got its start in Nutch. A few of us were attempting to build an open source web search engine and having trouble managing computations running on even a handful of computers. Once Google published its GFS and MapReduce papers, the route became clear. They’d devised systems to solve precisely the problems we were having with Nutch. So we started, two of us, half-time, to try to re-create these systems as a part of Nutch.

We managed to get Nutch limping along on 20 machines, but it soon became clear that to handle the Web’s massive scale, we’d need to run it on thousands of machines, and moreover, that the job was bigger than two half-time developers could handle.

Around that time, Yahoo! got interested, and quickly put together a team that I joined. We split off the distributed computing part of Nutch, naming it Hadoop. With the help of Yahoo!, Hadoop soon grew into a technology that could truly scale to the Web.

In 2006, Tom White started contributing to Hadoop. I already knew Tom through an excellent article he’d written about Nutch, so I knew he could present complex ideas in clear prose. I soon learned that he could also develop software that was as pleasant to read as his prose.

From the beginning, Tom’s contributions to Hadoop showed his concern for users and for the project. Unlike most open source contributors, Tom is not primarily interested in tweaking the system to better meet his own needs, but rather in ...