O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Elasticsearch 5.x Cookbook - Third Edition

Book Description

Over 170 advanced recipes to search, analyze, deploy, manage, and monitor data effectively with Elasticsearch 5.x

About This Book

  • Deploy and manage simple Elasticsearch nodes as well as complex cluster topologies
  • Write native plugins to extend the functionalities of Elasticsearch 5.x to boost your business
  • Packed with clear, step-by-step recipes to walk you through the capabilities of Elasticsearch 5.x

Who This Book Is For

If you are a developer who wants to get the most out of Elasticsearch for advanced search and analytics, this is the book for you. Some understanding of JSON is expected. If you want to extend Elasticsearch, understanding of Java and related technologies is also required.

What You Will Learn

  • Choose the best Elasticsearch cloud topology to deploy and power it up with external plugins
  • Develop tailored mapping to take full control of index steps
  • Build complex queries through managing indices and documents
  • Optimize search results through executing analytics aggregations
  • Monitor the performance of the cluster and nodes
  • Install Kibana to monitor cluster and extend Kibana for plugins
  • Integrate Elasticsearch in Java, Scala, Python and Big Data applications

In Detail

Elasticsearch is a Lucene-based distributed search server that allows users to index and search unstructured content with petabytes of data. This book is your one-stop guide to master the complete Elasticsearch ecosystem.

We'll guide you through comprehensive recipes on what's new in Elasticsearch 5.x, showing you how to create complex queries and analytics, and perform index mapping, aggregation, and scripting. Further on, you will explore the modules of Cluster and Node monitoring and see ways to back up and restore a snapshot of an index.

You will understand how to install Kibana to monitor a cluster and also to extend Kibana for plugins. Finally, you will also see how you can integrate your Java, Scala, Python, and Big Data applications such as Apache Spark and Pig with Elasticsearch, and add enhanced functionalities with custom plugins.

By the end of this book, you will have an in-depth knowledge of the implementation of the Elasticsearch architecture and will be able to manage data efficiently and effectively with Elasticsearch.

Style and approach

This book follows a problem-solution approach to effectively use and manage Elasticsearch. Each recipe focuses on a particular task at hand, and is explained in a very simple, easy to understand manner.

Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

Table of Contents

  1. Elasticsearch 5.x Cookbook Third Edition
    1. Credits
    2. About the Author
    3. About the Reviewer
    4. www.PacktPub.com
      1. eBooks, discount offers, and more
        1. Why subscribe?
    5. Customer Feedback
    6. Dedication
    7. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Sections
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. There's more…
        5. See also
      5. Conventions
      6. Reader feedback
      7. Customer support
      8. Downloading the example code
      9. Errata
      10. Piracy
      11. Questions
    8. 1. Getting Started
      1. Introduction
      2. Understanding node and cluster
        1. Getting ready
        2. How it work...
        3. There's more...
        4. See also
      3. Understanding node services
        1. Getting ready
        2. How it works...
      4. Managing your data
        1. Getting ready
        2. How it works...
        3. There's more...
          1. Best practices
        4. See also
      5. Understanding cluster, replication, and sharding
        1. Getting ready
        2. How it works...
          1. Best practice
        3. There's more...
          1. Solving the yellow status
          2. Solving the red status
        4. See also
      6. Communicating with Elasticsearch
        1. Getting ready
        2. How it works...
      7. Using the HTTP protocol
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      8. Using the native protocol
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
    9. 2. Downloading and Setup
      1. Introduction
      2. Downloading and installing Elasticsearch
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      3. Setting up networking
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      4. Setting up a node
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      5. Setting up for Linux systems
        1. Getting ready
        2. How to do it...
        3. How it works...
      6. Setting up different node types
        1. Getting ready
        2. How to do it...
        3. How it works...
      7. Setting up a client node
        1. Getting ready
        2. How to do it...
        3. How it works...
      8. Setting up an ingestion node
        1. Getting ready
        2. How to do it...
        3. How it works...
      9. Installing plugins in Elasticsearch
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      10. Installing plugins manually
        1. Getting ready
        2. How to do it...
        3. How it works...
      11. Removing a plugin
        1. Getting ready
        2. How to do it...
        3. How it works...
      12. Changing logging settings
        1. Getting ready
        2. How to do it...
        3. How it works...
      13. Setting up a node via Docker
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
    10. 3. Managing Mappings
      1. Introduction
      2. Using explicit mapping creation
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      3. Mapping base types
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      4. Mapping arrays
        1. Getting ready
        2. How to do it...
        3. How it works...
      5. Mapping an object
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      6. Mapping a document
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      7. Using dynamic templates in document mapping
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      8. Managing nested objects
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      9. Managing child document
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      10. Adding a field with multiple mapping
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      11. Mapping a GeoPoint field
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      12. Mapping a GeoShape field
        1. Getting ready
        2. How to do it
        3. How it works...
        4. See also
      13. Mapping an IP field
        1. Getting ready
        2. How to do it...
        3. How it works...
      14. Mapping an attachment field
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      15. Adding metadata to a mapping
        1. Getting ready
        2. How to do it...
        3. How it works...
      16. Specifying a different analyzer
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      17. Mapping a completion field
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
    11. 4. Basic Operations
      1. Introduction
      2. Creating an index
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      3. Deleting an index
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      4. Opening/closing an index
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      5. Putting a mapping in an index
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      6. Getting a mapping
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      7. Reindexing an index
        1. Getting ready
        2. How to do it...
        3. How it works...
      8. See also
      9. Refreshing an index
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      10. Flushing an index
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      11. ForceMerge an index
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      12. Shrinking an index
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      13. Checking if an index or type exists
        1. Getting ready
        2. How to do it...
        3. How it works...
      14. Managing index settings
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      15. Using index aliases
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      16. Rollover an index
        1. Getting ready
        2. How to do it…
        3. How it works...
        4. See also
      17. Indexing a document
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      18. Getting a document
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There is more...
        5. See also
      19. Deleting a document
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      20. Updating a document
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      21. Speeding up atomic operations (bulk operations)
        1. Getting ready
        2. How to do it...
        3. How it works...
      22. Speeding up GET operations (multi GET)
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also...
    12. 5. Search
      1. Introduction
      2. Executing a search
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      3. Sorting results
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      4. Highlighting results
        1. Getting ready
        2. How to do it...
        3. How it works…
        4. See also
      5. Executing a scrolling query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      6. Using the search_after functionality
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      7. Returning inner hits in results
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      8. Suggesting a correct query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      9. Counting matched results
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      10. Explaining a query
        1. Getting ready
        2. How to do it...
        3. How it works...
      11. Query profiling
        1. Getting ready
        2. How to do it...
        3. How it works...
      12. Deleting by query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      13. Updating by query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      14. Matching all the documents
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      15. Using a boolean query
        1. Getting ready
        2. How to do it...
        3. How it works...
    13. 6. Text and Numeric Queries
      1. Introduction
      2. Using a term query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      3. Using a terms query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      4. Using a prefix query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      5. Using a wildcard query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      6. Using a regexp query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      7. Using span queries
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      8. Using a match query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      9. Using a query string query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      10. Using a simple query string query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      11. Using the range query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      12. The common terms query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      13. Using IDs query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      14. Using the function score query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      15. Using the exists query
        1. Getting ready
        2. How to do it...
        3. How it works...
      16. Using the template query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
    14. 7. Relationships and Geo Queries
      1. Introduction
      2. Using the has_child query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      3. Using the has_parent query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      4. Using nested queries
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      5. Using the geo_bounding_box query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      6. Using the geo_polygon query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      7. Using the geo_distance query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      8. Using the geo_distance_range query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
    15. 8. Aggregations
      1. Introduction
      2. Executing an aggregation
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      3. Executing stats aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      4. Executing terms aggregation
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      5. Executing significant terms aggregation
        1. Getting ready
        2. How to do it...
        3. How it works...
      6. Executing range aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      7. Executing histogram aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      8. Executing date histogram aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      9. Executing filter aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      10. Executing filters aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
      11. Executing global aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
      12. Executing geo distance aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      13. Executing children aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
      14. Executing nested aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      15. Executing top hit aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      16. Executing a matrix stats aggregation
        1. Getting ready
        2. How to do it...
        3. How it works...
      17. Executing geo bounds aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      18. Executing geo centroid aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
    16. 9. Scripting
      1. Introduction
      2. Painless scripting
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more
        5. See also
      3. Installing additional script plugins
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      4. Managing scripts
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      5. Sorting data using scripts
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      6. Computing return fields with scripting
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      7. Filtering a search via scripting
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      8. Using scripting in aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
      9. Updating a document using scripts
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      10. Reindexing with a script
        1. Getting ready
        2. How to do it...
        3. How it works...
    17. 10. Managing Clusters and Nodes
    18. 11. Backup and Restore
      1. Introduction
      2. Managing repositories
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      3. Executing a snapshot
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      4. Restoring a snapshot
        1. Getting ready
        2. How to do it...
        3. How it works...
      5. Setting up a NFS share for backup
        1. Getting ready
        2. How to do it...
        3. How it works...
      6. Reindexing from a remote cluster
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
    19. 12. User Interfaces
      1. Introduction
      2. Installing and using Cerebro
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      3. Installing Kibana and X-Pack
        1. Getting ready
        2. How to do it...
        3. How it works...
      4. Managing Kibana dashboards
        1. Getting ready
        2. How to do it...
        3. How it works...
      5. Monitoring with Kibana
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      6. Using Kibana dev-console
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      7. Visualizing data with Kibana
        1. Getting ready
        2. How to do it...
        3. How it works...
      8. Installing Kibana plugins
        1. Getting ready
        2. How to do it...
        3. How it works...
      9. Generating graph with Kibana
        1. Getting ready
        2. How to do it...
        3. How it works...
    20. 13. Ingest
      1. Introduction
      2. Pipeline definition
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      3. Put an ingest pipeline
        1. Getting ready
        2. How to do it...
        3. How it works...
      4. Get an ingest pipeline
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      5. Delete an ingest pipeline
        1. Getting ready
        2. How to do it...
        3. How it works...
      6. Simulate an ingest pipeline
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      7. Built-in processors
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      8. Grok processor
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      9. Using the ingest attachment plugin
        1. Getting ready
        2. How to do it...
        3. How it works...
      10. Using the ingest GeoIP plugin
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
    21. 14. Java Integration
      1. Introduction
      2. Creating a standard Java HTTP client
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      3. Creating an HTTP Elasticsearch client
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      4. Creating a native client
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      5. Managing indices with the native client
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      6. Managing mappings
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      7. Managing documents
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      8. Managing bulk actions
        1. Getting ready
        2. How to do it...
        3. How it works...
      9. Building a query
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      10. Executing a standard search
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      11. Executing a search with aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      12. Executing a scroll search
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
    22. 15. Scala Integration
      1. Introduction
      2. Creating a client in Scala
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      3. Managing indices
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      4. Managing mappings
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      5. Managing documents
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
        5. See also
      6. Executing a standard search
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      7. Executing a search with aggregations
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
    23. 16. Python Integration
      1. Introduction
      2. Creating a client
        1. Getting ready
        2. How to do it...
        3. How it works…
        4. See also
      3. Managing indices
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. There's more…
        5. See also
      4. Managing mappings include the mapping
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. See also
      5. Managing documents
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. See also
      6. Executing a standard search
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. See also
      7. Executing a search with aggregations
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. See also
    24. 17. Plugin Development
      1. Introduction
      2. Creating a plugin
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      3. Creating an analyzer plugin
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      4. Creating a REST plugin
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      5. Creating a cluster action
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      6. Creating an ingest plugin
        1. Getting ready
        2. How to do it...
        3. How it works...
    25. 18. Big Data Integration
      1. Introduction
      2. Installing Apache Spark
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      3. Indexing data via Apache Spark
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. See also
      4. Indexing data with meta via Apache Spark
        1. Getting ready
        2. How to do it...
        3. How it works...
        4. There's more...
      5. Reading data with Apache Spark
        1. Getting ready
        2. How to do it...
        3. How it works...
      6. Reading data using SparkSQL
        1. Getting ready
        2. How to do it...
        3. How it works...
      7. Indexing data with Apache Pig
        1. Getting ready
        2. How to do it...
        3. How it works...