book

HBase: The Definitive Guide

by Lars George

September 2011

Intermediate to advanced

552 pages

16h 31m

English

O'Reilly Media, Inc.

Read now

Unlock full access

General InformationHBase VersionBuilding the ExamplesHush: The HBase URL ShortenerRunning HushConventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
The Dawn of Big DataThe Problem with Relational Database SystemsNonrelational Database Systems, Not-Only SQL or NoSQL?DimensionsScalabilityDatabase (De-)NormalizationBuilding BlocksBackdropTables, Rows, Columns, and CellsAuto-ShardingStorage APIImplementationSummaryHBase: The Hadoop DatabaseHistoryNomenclatureSummary
Quick-Start GuideRequirementsHardwareServersNetworkingSoftwareOperating systemFilesystemJavaHadoopSSHDomain Name ServiceSynchronized timeFile handles and process limitsDatanode handlersSwappinessWindowsFilesystems for HBaseLocalHDFSS3Other FilesystemsInstallation ChoicesApache Binary ReleaseBuilding from SourceRun ModesStandalone ModeDistributed ModePseudodistributed modeFully distributed modeSpecifying region serversZooKeeper setupUsing the existing ZooKeeper ensembleConfigurationhbase-site.xml and hbase-default.xmlhbase-env.shregionserverlog4j.propertiesExample Configurationhbase-site.xmlregionservershbase-env.shClient ConfigurationDeploymentScript-BasedApache WhirrPuppet and ChefOperating a ClusterRunning and Confirming Your InstallationWeb-based UI IntroductionShell IntroductionStopping the Cluster
General NotesCRUD OperationsPut MethodSingle PutsThe KeyValue classClient-side write bufferList of PutsAtomic compare-and-setGet MethodSingle GetsThe Result classList of GetsRelated retrieval methodsDelete MethodSingle DeletesList of DeletesAtomic compare-and-deleteBatch OperationsRow LocksScansIntroductionThe ResultScanner ClassCaching Versus BatchingMiscellaneous FeaturesThe HTable Utility MethodsThe Bytes Class
FiltersIntroduction to FiltersThe filter hierarchyComparison operatorsComparatorsComparison FiltersRowFilterFamilyFilterQualifierFilterValueFilterDependentColumnFilterDedicated FiltersSingleColumnValueFilterSingleColumnValueExcludeFilterPrefixFilterPageFilterKeyOnlyFilterFirstKeyOnlyFilterInclusiveStopFilterTimestampsFilterColumnCountGetFilterColumnPaginationFilterColumnPrefixFilterRandomRowFilterDecorating FiltersSkipFilterWhileMatchFilterFilterListCustom FiltersFilters SummaryCountersIntroduction to CountersSingle CountersMultiple CountersCoprocessorsIntroduction to CoprocessorsThe Coprocessor ClassCoprocessor LoadingLoading from the configurationLoading from the table descriptorThe RegionObserver ClassHandling region life-cycle eventsState: pending openState: openState: pending closeHandling client API eventsThe RegionCoprocessorEnvironment classThe ObserverContext classThe BaseRegionObserver classThe MasterObserver ClassThe MasterCoprocessorEnvironment classThe BaseMasterObserver classEndpointsThe CoprocessorProtocol interfaceThe BaseEndpointCoprocessor classHTablePoolConnection Handling
Schema DefinitionTablesTable PropertiesColumn FamiliesHBaseAdminBasic OperationsTable OperationsSchema OperationsCluster OperationsCluster Status Information
Introduction to REST, Thrift, and AvroInteractive ClientsNative JavaRESTOperationSupported formatsPlain (text/plain)XML (text/xml)JSON (application/json)Protocol Buffer (application/x-protobuf)Raw binary (application/octet-stream)REST Java clientThriftInstallationOperationExample: PHPAvroInstallationOperationOther ClientsBatch ClientsMapReduceNative JavaClojureHivePigCascadingShellBasicsCommandsGeneralData definitionData manipulationToolsReplicationScriptingWeb-based UIMaster UIMain pageUser Table pageZooKeeper pageRegion Server UIMain pageShared Pages
FrameworkMapReduce IntroductionClassesInputFormatMapperReducerOutputFormatSupporting ClassesMapReduce LocalityTable SplitsMapReduce over HBasePreparationStatic ProvisioningDynamic ProvisioningData SinkData SourceData Source and SinkCustom Processing

Seek Versus TransferB+ TreesLog-Structured Merge-TreesStorageOverviewWrite PathFilesRoot-level filesTable-level filesRegion-level filesRegion splitsCompactionsHFile FormatKeyValue FormatWrite-Ahead LogOverviewHLog ClassHLogKey ClassWALEdit ClassLogSyncer ClassLogRoller ClassReplaySingle logLog splittingEdits recoveryDurabilityRead PathRegion LookupsThe Region Life CycleZooKeeperReplicationLife of a Log EditNormal processingNon-Responding slave clustersInternalsChoosing region servers to replicate toKeeping track of logsReading, filtering, and sending editsCleaning logsRegion server failover
Key DesignConceptsTall-Narrow Versus Flat-Wide TablesPartial Key ScansPaginationTime Series DataTime-Ordered RelationsAdvanced SchemasSecondary IndexesSearch IntegrationTransactionsBloom FiltersVersioningImplicit VersioningCustom Versioning
IntroductionThe Metrics FrameworkContexts, Records, and MetricsMaster MetricsRegion Server MetricsRPC MetricsJVM MetricsInfo MetricsGangliaInstallationGanglia-related stepsGanglia monitoring daemonGanglia meta daemonGanglia web frontendHBase-related stepsUsageJMXJConsoleJMX Remote APINagios
Garbage Collection TuningMemstore-Local Allocation BufferCompressionAvailable CodecsSnappyLZOGZIPVerifying InstallationCompression test toolStartup checkEnabling CompressionOptimizing Splits and CompactionsManaged SplittingRegion HotspottingPresplitting RegionsLoad BalancingMerging RegionsClient API: Best PracticesConfigurationLoad TestsPerformance EvaluationYCSB
Operational TasksNode DecommissioningRolling RestartsAdding ServersPseudodistributed modeAdding a local backup masterAdding a local region serverFully distributed clusterAdding a backup masterAdding a region serverData TasksImport and Export ToolsCopyTable ToolBulk ImportBulk load procedureUsing the importtsv toolUsing the completebulkload ToolAdvanced usageReplicationAdditional TasksCoexisting ClustersRequired PortsChanging Logging LevelsTroubleshootingHBase FsckAnalyzing the LogsCommon IssuesBasic setup checklistFile handlesDataNode connectionsCompressionGarbage collection/memory tuningStability issuesZooKeeper problems“Could not obtain block” errors
HBase 0.92.0HBase 0.94.0
Upgrading to HBase 0.90.xFrom 0.20.x or 0.89.xWithin 0.90.xUpgrading to HBase 0.92.0
Cloudera’s Distribution Including Apache Hadoop

Content preview from HBase: The Definitive Guide

Appendix D. Distributions

There are more choices to install HBase than using the Apache releases. Here we list what is available alternatively.

Cloudera’s Distribution Including Apache Hadoop

Cloudera’s Distribution including Apache Hadoop (hereafter CDH) is based on the most recent stable version of Apache Hadoop with numerous patches, backports, and updates. Cloudera makes the distribution available in a number of different formats: source and binary tar files, RPMs, Debian packages, VMware images, and scripts for running CDH in the cloud. CDH is free, released under the Apache 2.0 license and available at http://www.cloudera.com/hadoop/.

To simplify deployment, Cloudera hosts packages on public yum and apt repositories. CDH enables you to install and configure Hadoop, and HBase, on each machine using a single command. Kickstart users can commission entire Hadoop clusters without manual intervention.

CDH manages cross-component versions and provides a stable platform with a compatible set of packages that work together. As of CDH3, the following packages are included, many of which are covered elsewhere in this book:

HDFS: Self-healing distributed filesystem
MapReduce: Powerful, parallel data processing framework
Hadoop Common: A set of utilities that support the Hadoop subprojects
HBase: Hadoop database for random read/write access
Hive: SQL-like queries and tables on large data sets
Pig: Dataflow language and compiler
Oozie: Workflow for interdependent Hadoop jobs
Sqoop: Integrates databases and data warehouses ...