book

Real-World Hadoop

by Ellen Friedman, Ted Dunning

April 2015

Beginner to intermediate

104 pages

2h 28m

English

O'Reilly Media, Inc.

Read now

Unlock full access

How to Use This Book
A Day in the Life of a Big Data ProjectFrom Aadhaar to Your Own Big Data ProjectWhat Hadoop and NoSQL DoWhen Are Hadoop and NoSQL the Right Choice?
Typical FunctionsData Storage and PersistenceData IngestApache KafkaApache SqoopApache FlumeData Extraction from HadoopProcessing, Transforming, QueryingStreamingMicro-batchingBatch ProcessingInteractive QueryImpalaApache DrillApache SparkSearch Abuse—Using Search and Indexing for Interactive QueryVisualization ToolsIntegration via ODBC and JDBC
Use of Existing Non-Hadoop ApplicationsMaking Use of a Realtime Distributed File SystemMeeting SLAsDeploying Data at Scale to Remote LocationsConsistent Data VersioningFinding the Keys to Success
Tip #1: Pick One Thing to Do FirstTip #2: Shift Your ThinkingTip #3: Start Conservatively But Plan to ExpandTip #4: Be Honest with YourselfTip #5: Plan Ahead for MaintenanceTip #6: Think Big: Don’t Underestimate What You Can (and Will) Want to DoTip #7: Explore New Data FormatsTip #8: Consider Data Placement When You Expand a ClusterTip #9: Plot Your ExpansionTip #10: Form a Queue to the Right, PleaseTip #11: Provide Reliable Primary Persistence When Using Search ToolsTip #12: Establish Remote Clusters for Disaster RecoveryTip #13: Take a Complete View of PerformanceTip #14: Read Our Other Books (Really!)Tip # 15: Just Do It
Data Warehouse OptimizationData HubCustomer 360Recommendation EngineMarketing OptimizationLarge Object StoreLog ProcessingRealtime AnalyticsTime Series Database
TelecomsWhat Customers WantWorking with MoneySensor Data, Predictive Maintenance, and a “Time Machine”A Time MachineManufacturingExtending Quality Assurance
Additional Publications

Content preview from Real-World Hadoop

Appendix A. Additional Resources

The following open source Apache Foundation projects are the inspiration for the revolution described in this book.

Apache Hadoop: A distributed computing system for large-scale data
Apache HBase: A non-relational NoSQL database that runs on Hadoop

The following projects provide core tools among those described in this book.

Apache Drill: A flexible, ad hoc SQL-on-Hadoop query engine that can use nested data
Apache Hive: A SQL-like query engine, the first to provide this type of approach for Hadoop
Apache Spark: An in-memory query processing tool that includes a realtime processing component
Apache Storm: Realtime stream processing tool
Apache Kafka: Message-queuing system
Apache Solr: Search technology based on Apache Lucene
ElasticSearch: Search technology based on Apache Lucene

The use cases described in this book are based on the Hadoop distribution from MapR Technologies.

For cluster validation, there is a github repository that contains a variety of preinstallation tests that are used by MapR to verify correct hardware operation. Since these are preinstallation tests, they can be used to validate clusters before installing other Hadoop distributions as well.

Additional Publications

The authors have also written these short books published by O’Reilly that provide additional detail about some of the techniques mentioned in the Hadoop and NoSQL use cases covered in this book.