O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Scaling Big Data with Hadoop and Solr

Book Description

By combining Apache Hadoop and Solr you can build super-efficient, high-speed enterprise search engines, and this book takes you through every stage of the process with a practical tutorial. Written specifically for Java programmers.

  • Understand the different approaches of making Solr work on Big Data as well as the benefits and drawbacks
  • Learn from interesting, real-life use cases for Big Data search along with sample code
  • Work with the Distributed Enterprise Search without prior knowledge of Hadoop and Solr

In Detail

As data grows exponentially day-by-day, extracting information becomes a tedious activity in itself. Technologies like Hadoop are trying to address some of the concerns, while Solr provides high-speed faceted search. Bringing these two technologies together is helping organizations resolve the problem of information extraction from Big Data by providing excellent distributed faceted search capabilities.

Scaling Big Data with Hadoop and Solr is a step-by-step guide that helps you build high performance enterprise search engines while scaling data. Starting with the basics of Apache Hadoop and Solr, this book then dives into advanced topics of optimizing search with some interesting real-world use cases and sample Java code.

Scaling Big Data with Hadoop and Solr starts by teaching you the basics of Big Data technologies including Hadoop and its ecosystem and Apache Solr. It explains the different approaches of scaling Big Data with Hadoop and Solr, with discussion regarding the applicability, benefits, and drawbacks of each approach. It then walks readers through how sharding and indexing can be performed on Big Data followed by the performance optimization of Big Data search. Finally, it covers some real-world use cases for Big Data scaling.

With this book, you will learn everything you need to know to build a distributed enterprise search platform as well as how to optimize this search to a greater extent resulting in maximum utilization of available resources.

Table of Contents

  1. Scaling Big Data with Hadoop and Solr
    1. Table of Contents
    2. Scaling Big Data with Hadoop and Solr
    3. Credits
    4. About the Author
    5. About the Reviewer
    6. www.PacktPub.com
      1. Support files, eBooks, discount offers and more
        1. Why Subscribe?
        2. Free Access for Packt account holders
    7. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the example code
        2. Errata
        3. Piracy
        4. Questions
    8. 1. Processing Big Data Using Hadoop and MapReduce
      1. Understanding Apache Hadoop and its ecosystem
        1. The ecosystem of Apache Hadoop
          1. Apache HBase
          2. Apache Pig
          3. Apache Hive
          4. Apache ZooKeeper
          5. Apache Mahout
          6. Apache HCatalog
          7. Apache Ambari
          8. Apache Avro
          9. Apache Sqoop
          10. Apache Flume
      2. Storing large data in HDFS
        1. HDFS architecture
          1. NameNode
          2. DataNode
          3. Secondary NameNode
        2. Organizing data
        3. Accessing HDFS
      3. Creating MapReduce to analyze Hadoop data
        1. MapReduce architecture
          1. JobTracker
          2. TaskTracker
      4. Installing and running Hadoop
        1. Prerequisites
        2. Setting up SSH without passphrases
        3. Installing Hadoop on machines
        4. Hadoop configuration
        5. Running a program on Hadoop
      5. Managing a Hadoop cluster
      6. Summary
    9. 2. Understanding Solr
      1. Installing Solr
      2. Apache Solr architecture
        1. Storage
        2. Solr engine
          1. The query parser
          2. Interaction
          3. Client APIs and SolrJ client
          4. Other interfaces
      3. Configuring Apache Solr search
        1. Defining a Schema for your instance
        2. Configuring a Solr instance
          1. Configuration files
        3. Request handlers and search components
          1. Facet
          2. MoreLikeThis
          3. Highlight
          4. SpellCheck
          5. Metadata management
      4. Loading your data for search
        1. ExtractingRequestHandler/Solr Cell
        2. SolrJ
      5. Summary
    10. 3. Making Big Data Work for Hadoop and Solr
      1. The problem
      2. Understanding data-processing workflows
        1. The standalone machine
        2. Distributed setup
        3. The replicated mode
        4. The sharded mode
      3. Using Solr 1045 patch – map-side indexing
        1. Benefits and drawbacks
          1. Benefits
          2. Drawbacks
      4. Using Solr 1301 patch – reduce-side indexing
        1. Benefits and drawbacks
          1. Benefits
          2. Drawbacks
      5. Using SolrCloud for distributed search
        1. SolrCloud architecture
        2. Configuring SolrCloud
        3. Using multicore Solr search on SolrCloud
        4. Benefits and drawbacks
          1. Benefits
          2. Drawbacks
      6. Using Katta for Big Data search (Solr-1395 patch)
        1. Katta architecture
        2. Configuring Katta cluster
        3. Creating Katta indexes
        4. Benefits and drawbacks
          1. Benefits
          2. Drawbacks
      7. Summary
    11. 4. Using Big Data to Build Your Large Indexing
      1. Understanding the concept of NOSQL
      2. The CAP theorem
        1. What is a NOSQL database?
          1. The key-value store or column store
          2. The document-oriented store
          3. The graph database
        2. Why NOSQL databases for Big Data?
        3. How Solr can be used for Big Data storage?
      3. Understanding the concepts of distributed search
        1. Distributed search architecture
        2. Distributed search scenarios
      4. Lily – running Solr and Hadoop together
        1. The architecture
          1. Write-ahead Logging
          2. The message queue
          3. Querying using Lily
          4. Updating records using Lily
        2. Installing and running Lily
      5. Deep dive – shards and indexing data of Apache Solr
        1. The sharding algorithm
        2. Adding a document to the distributed shard
      6. Configuring SolrCloud to work with large indexes
        1. Setting up the ZooKeeper ensemble
        2. Setting up the Apache Solr instance
        3. Creating shards, collections, and replicas in SolrCloud
      7. Summary
    12. 5. Improving Performance of Search while Scaling with Big Data
      1. Understanding the limits
      2. Optimizing the search schema
        1. Specifying the default search field
        2. Configuring search schema fields
        3. Stop words
        4. Stemming
      3. Index optimization
        1. Limiting the indexing buffer size
        2. When to commit changes?
        3. Optimizing the index merge
        4. Optimize an option for index merging
        5. Optimizing the container
        6. Optimizing concurrent clients
        7. Optimizing the Java virtual memory
      4. Optimization the search runtime
        1. Optimizing through search queries
          1. Filter queries
        2. Optimizing the Solr cache
          1. The filter cache
          2. The query result cache
          3. The document cache
          4. The field value cache
          5. Lazy field loading
        3. Optimizing search on Hadoop
      5. Monitoring the Solr instance
        1. Using SolrMeter
      6. Summary
    13. A. Use Cases for Big Data Search
      1. E-commerce websites
      2. Log management for banking
        1. The problem
        2. How can it be tackled?
        3. High-level design
    14. B. Creating Enterprise Search Using Apache Solr
      1. schema.xml
      2. solrconfig.xml
      3. spellings.txt
      4. synonyms.txt
      5. protwords.txt
      6. stopwords.txt
    15. C. Sample MapReduce Programs to Build the Solr Indexes
      1. The Solr-1045 patch – map program
      2. The Solr-1301 patch – reduce-side indexing
      3. Katta
    16. Index