book

Hadoop: The Definitive Guide, 4th Edition

by Tom White

April 2015

Beginner to intermediate

754 pages

21h 39m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Administrative NotesWhat’s New in the Fourth Edition?What’s New in the Third Edition?What’s New in the Second Edition?Conventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
Data!Data Storage and AnalysisQuerying All Your DataBeyond BatchComparison with Other SystemsRelational Database Management SystemsGrid ComputingVolunteer ComputingA Brief History of Apache HadoopWhat’s in This Book?
A Weather DatasetData FormatAnalyzing the Data with Unix ToolsAnalyzing the Data with HadoopMap and ReduceJava MapReduceA test runScaling OutData FlowCombiner FunctionsSpecifying a combiner functionRunning a Distributed MapReduce JobHadoop StreamingRubyPython
The Design of HDFSHDFS ConceptsBlocksNamenodes and DatanodesBlock CachingHDFS FederationHDFS High AvailabilityFailover and fencingThe Command-Line InterfaceBasic Filesystem OperationsHadoop FilesystemsInterfacesHTTPCNFSFUSEThe Java InterfaceReading Data from a Hadoop URLReading Data Using the FileSystem APIFSDataInputStreamWriting DataFSDataOutputStreamDirectoriesQuerying the FilesystemFile metadata: FileStatusListing filesFile patternsPathFilterDeleting DataData FlowAnatomy of a File ReadAnatomy of a File WriteCoherency ModelConsequences for application designParallel Copying with distcpKeeping an HDFS Cluster Balanced
Anatomy of a YARN Application RunResource RequestsApplication LifespanBuilding YARN ApplicationsYARN Compared to MapReduce 1Scheduling in YARNScheduler OptionsCapacity Scheduler ConfigurationQueue placementFair Scheduler ConfigurationEnabling the Fair SchedulerQueue configurationQueue placementPreemptionDelay SchedulingDominant Resource FairnessFurther Reading
Data IntegrityData Integrity in HDFSLocalFileSystemChecksumFileSystemCompressionCodecsCompressing and decompressing streams with CompressionCodecInferring CompressionCodecs using CompressionCodecFactoryNative librariesCodecPoolCompression and Input SplitsUsing Compression in MapReduceCompressing map outputSerializationThe Writable InterfaceWritableComparable and comparatorsWritable ClassesWritable wrappers for Java primitivesTextIndexingUnicodeIterationMutabilityResorting to StringBytesWritableNullWritableObjectWritable and GenericWritableWritable collectionsImplementing a Custom WritableImplementing a RawComparator for speedCustom comparatorsSerialization FrameworksSerialization IDLFile-Based Data StructuresSequenceFileWriting a SequenceFileReading a SequenceFileDisplaying a SequenceFile with the command-line interfaceSorting and merging SequenceFilesThe SequenceFile formatMapFileMapFile variantsOther File Formats and Column-Oriented Formats

The Configuration APICombining ResourcesVariable ExpansionSetting Up the Development EnvironmentManaging ConfigurationGenericOptionsParser, Tool, and ToolRunnerWriting a Unit Test with MRUnitMapperReducerRunning Locally on Test DataRunning a Job in a Local Job RunnerTesting the DriverRunning on a ClusterPackaging a JobThe client classpathThe task classpathPackaging dependenciesTask classpath precedenceLaunching a JobThe MapReduce Web UIThe resource manager pageThe MapReduce job pageRetrieving the ResultsDebugging a JobThe tasks and task attempts pagesHandling malformed dataHadoop LogsRemote DebuggingTuning a JobProfiling TasksThe HPROF profilerMapReduce WorkflowsDecomposing a Problem into MapReduce JobsJobControlApache OozieDefining an Oozie workflowPackaging and deploying an Oozie workflow applicationRunning an Oozie workflow job
Anatomy of a MapReduce Job RunJob SubmissionJob InitializationTask AssignmentTask ExecutionStreamingProgress and Status UpdatesJob CompletionFailuresTask FailureApplication Master FailureNode Manager FailureResource Manager FailureShuffle and SortThe Map SideThe Reduce SideConfiguration TuningTask ExecutionThe Task Execution EnvironmentStreaming environment variablesSpeculative ExecutionOutput CommittersTask side-effect files
MapReduce TypesThe Default MapReduce JobThe default Streaming jobKeys and values in StreamingInput FormatsInput Splits and RecordsFileInputFormatFileInputFormat input pathsFileInputFormat input splitsSmall files and CombineFileInputFormatPreventing splittingFile information in the mapperProcessing a whole file as a recordText InputTextInputFormatControlling the maximum line lengthKeyValueTextInputFormatNLineInputFormatXMLBinary InputSequenceFileInputFormatSequenceFileAsTextInputFormatSequenceFileAsBinaryInputFormatFixedLengthInputFormatMultiple InputsDatabase Input (and Output)Output FormatsText OutputBinary OutputSequenceFileOutputFormatSequenceFileAsBinaryOutputFormatMapFileOutputFormatMultiple OutputsAn example: Partitioning dataMultipleOutputsLazy OutputDatabase Output
CountersBuilt-in CountersTask countersJob countersUser-Defined Java CountersDynamic countersRetrieving countersUser-Defined Streaming CountersSortingPreparationPartial SortTotal SortSecondary SortJava codeStreamingJoinsMap-Side JoinsReduce-Side JoinsSide Data DistributionUsing the Job ConfigurationDistributed CacheUsageHow it worksThe distributed cache APIMapReduce Library Classes
Cluster SpecificationCluster SizingMaster node scenariosNetwork TopologyRack awarenessCluster Setup and InstallationInstalling JavaCreating Unix User AccountsInstalling HadoopConfiguring SSHConfiguring HadoopFormatting the HDFS FilesystemStarting and Stopping the DaemonsCreating User DirectoriesHadoop ConfigurationConfiguration ManagementEnvironment SettingsJavaMemory heap sizeSystem logfilesSSH settingsImportant Hadoop Daemon PropertiesHDFSYARNMemory settings in YARN and MapReduceCPU settings in YARN and MapReduceHadoop Daemon Addresses and PortsOther Hadoop PropertiesCluster membershipBuffer sizeHDFS block sizeReserved storage spaceTrashJob schedulerReduce slow startShort-circuit local readsSecurityKerberos and HadoopAn exampleDelegation TokensOther Security EnhancementsBenchmarking a Hadoop ClusterHadoop BenchmarksBenchmarking MapReduce with TeraSortOther benchmarksUser Jobs
HDFSPersistent Data StructuresNamenode directory structureThe filesystem image and edit logSecondary namenode directory structureDatanode directory structureSafe ModeEntering and leaving safe modeAudit LoggingToolsdfsadminFilesystem check (fsck)Finding the blocks for a fileDatanode block scannerBalancerMonitoringLoggingSetting log levelsGetting stack tracesMetrics and JMXMaintenanceRoutine Administration ProceduresMetadata backupsData backupsFilesystem check (fsck)Filesystem balancerCommissioning and Decommissioning NodesCommissioning new nodesDecommissioning old nodesUpgradesHDFS data and metadata upgradesStart the upgradeWait until the upgrade is completeCheck the upgradeRoll back the upgrade (optional)Finalize the upgrade (optional)
Avro Data Types and SchemasIn-Memory Serialization and DeserializationThe Specific APIAvro DatafilesInteroperabilityPython APIAvro ToolsSchema ResolutionSort OrderAvro MapReduceSorting Using Avro MapReduceAvro in Other Languages
Data ModelNested EncodingParquet File FormatParquet ConfigurationWriting and Reading Parquet FilesAvro, Protocol Buffers, and ThriftProjection and read schemasParquet MapReduce
Installing FlumeAn ExampleTransactions and ReliabilityBatchingThe HDFS SinkPartitioning and InterceptorsFile FormatsFan OutDelivery GuaranteesReplicating and Multiplexing SelectorsDistribution: Agent TiersDelivery GuaranteesSink GroupsIntegrating Flume with ApplicationsComponent CatalogFurther Reading
Getting SqoopSqoop ConnectorsA Sample ImportText and Binary File FormatsGenerated CodeAdditional Serialization SystemsImports: A Deeper LookControlling the ImportImports and ConsistencyIncremental ImportsDirect-Mode ImportsWorking with Imported DataImported Data and HiveImporting Large ObjectsPerforming an ExportExports: A Deeper LookExports and TransactionalityExports and SequenceFilesFurther Reading
Installing and Running PigExecution TypesLocal modeMapReduce modeRunning Pig ProgramsGruntPig Latin EditorsAn ExampleGenerating ExamplesComparison with DatabasesPig LatinStructureStatementsExpressionsTypesSchemasUsing Hive tables with HCatalogValidation and nullsSchema mergingFunctionsOther librariesMacrosUser-Defined FunctionsA Filter UDFLeveraging typesAn Eval UDFDynamic invokersA Load UDFUsing a schemaData Processing OperatorsLoading and Storing DataFiltering DataFOREACH...GENERATESTREAMGrouping and Joining DataJOINCOGROUPCROSSGROUPSorting DataCombining and Splitting DataPig in PracticeParallelismAnonymous RelationsParameter SubstitutionDynamic parametersParameter substitution processingFurther Reading
Installing HiveThe Hive ShellAn ExampleRunning HiveConfiguring HiveExecution enginesLoggingHive ServicesHive clientsThe MetastoreComparison with Traditional DatabasesSchema on Read Versus Schema on WriteUpdates, Transactions, and IndexesSQL-on-Hadoop AlternativesHiveQLData TypesPrimitive typesComplex typesOperators and FunctionsConversionsTablesManaged Tables and External TablesPartitions and BucketsPartitionsBucketsStorage FormatsThe default storage format: Delimited textBinary storage formats: Sequence files, Avro datafiles, Parquet files, RCFiles, and ORCFilesUsing a custom SerDe: RegexSerDeStorage handlersImporting DataInsertsMultitable insertCREATE TABLE...AS SELECTAltering TablesDropping TablesQuerying DataSorting and AggregatingMapReduce ScriptsJoinsInner joinsOuter joinsSemi joinsMap joinsSubqueriesViewsUser-Defined FunctionsWriting a UDFWriting a UDAFA more complex UDAFFurther Reading
An ExampleThe Core Crunch APIPrimitive Operationsunion()parallelDo()groupByKey()combineValues()TypesRecords and tuplesSources and TargetsReading from a sourceWriting to a targetExisting outputsCombined sources and targetsFunctionsSerialization of functionsObject reuseMaterializationPObjectPipeline ExecutionRunning a PipelineAsynchronous executionDebuggingStopping a PipelineInspecting a Crunch PlanIterative AlgorithmsCheckpointing a PipelineCrunch LibrariesFurther Reading
Installing SparkAn ExampleSpark Applications, Jobs, Stages, and TasksA Scala Standalone ApplicationA Java ExampleA Python ExampleResilient Distributed DatasetsCreationTransformations and ActionsAggregation transformationsPersistencePersistence levelsSerializationDataFunctionsShared VariablesBroadcast VariablesAccumulatorsAnatomy of a Spark Job RunJob SubmissionDAG ConstructionTask SchedulingTask ExecutionExecutors and Cluster ManagersSpark on YARNYARN client modeYARN cluster modeFurther Reading
HBasicsBackdropConceptsWhirlwind Tour of the Data ModelRegionsLockingImplementationHBase in operationInstallationTest DriveClientsJavaMapReduceREST and ThriftBuilding an Online Query ApplicationSchema DesignLoading DataLoad distributionBulk loadOnline QueriesStation queriesObservation queriesHBase Versus RDBMSSuccessful ServiceHBasePraxisHDFSUIMetricsCountersFurther Reading
Installing and Running ZooKeeperAn ExampleGroup Membership in ZooKeeperCreating the GroupJoining a GroupListing Members in a GroupZooKeeper command-line toolsDeleting a GroupThe ZooKeeper ServiceData ModelEphemeral znodesSequence numbersWatchesOperationsMultiupdateAPIsWatch triggersACLsImplementationConsistencySessionsTimeStatesBuilding Applications with ZooKeeperA Configuration ServiceThe Resilient ZooKeeper ApplicationInterruptedExceptionKeeperExceptionState exceptionsRecoverable exceptionsUnrecoverable exceptionsA reliable configuration serviceA Lock ServiceThe herd effectRecoverable exceptionsUnrecoverable exceptionsImplementationMore Distributed Data Structures and ProtocolsBookKeeper and HedwigZooKeeper in ProductionResilience and PerformanceConfigurationFurther Reading
From CPUs to Semantic IntegrationEnter Apache CrunchBuilding a Complete PictureIntegrating Healthcare DataComposability over FrameworksMoving Forward
The Structure of DNAThe Genetic Code: Turning DNA Letters into ProteinsThinking of DNA as Source CodeThe Human Genome Project and Reference GenomesSequencing and Aligning DNAADAM, A Scalable Genome Analysis PlatformLiterate programming with the Avro interface description language (IDL)Column-oriented access with ParquetA simple example: k-mer counting using Spark and ADAMFrom Personalized Ads to Personalized MedicineJoin In
Fields, Tuples, and PipesOperationsTaps, Schemes, and FlowsCascading in PracticeFlexibilityHadoop and Cascading at ShareThisSummary
PrerequisitesInstallationConfigurationStandalone ModePseudodistributed ModeConfiguring SSHFormatting the HDFS filesystemStarting and stopping the daemonsCreating a user directoryFully Distributed Mode

Content preview from Hadoop: The Definitive Guide, 4th Edition

Chapter 14. Flume

Hadoop is built for processing very large datasets. Often it is assumed that the data is already in HDFS, or can be copied there in bulk. However, there are many systems that don’t meet this assumption. They produce streams of data that we would like to aggregate, store, and analyze using Hadoop—and these are the systems that Apache Flume is an ideal fit for.

Flume is designed for high-volume ingestion into Hadoop of event-based data. The canonical example is using Flume to collect logfiles from a bank of web servers, then moving the log events from those files into new aggregated files in HDFS for processing. The usual destination (or sink in Flume parlance) is HDFS. However, Flume is flexible enough to write to other systems, like HBase or Solr.

To use Flume, we need to run a Flume agent, which is a long-lived Java process that runs sources and sinks, connected by channels. A source in Flume produces events and delivers them to the channel, which stores the events until they are forwarded to the sink. You can think of the source-channel-sink combination as a basic Flume building block.

A Flume installation is made up of a collection of connected agents running in a distributed topology. Agents on the edge of the system (co-located on web server machines, for example) collect data and forward it to agents that are responsible for aggregating and then storing the data in its final destination. Agents are configured to run a collection of particular sources and sinks, ...