book

Elasticsearch Server - Third Edition - Third Edition

Name: Elasticsearch Server - Third Edition - Third Edition
ISBN: 9781785888816

by Marek Rogozinski, Rafal Kuc

February 2016

Intermediate to advanced

556 pages

11h 53m

English

Packt Publishing

Read now

Unlock full access

Elasticsearch Server Third Edition
Table of Contents
Elasticsearch Server Third Edition
Credits
About the Authors
About the Reviewer
www.PacktPub.com
eBooks, discount offers, and moreWhy subscribe?
Preface
What this book covers
What you need for this book
Who this book is for

Conventions
Reader feedback
Customer support
Downloading the example codeDownloading the color images of this bookErrataPiracyQuestions
1. Getting Started with Elasticsearch Cluster
Full text searchingThe Lucene glossary and architectureInput data analysisIndexing and queryingScoring and query relevance
The basics of Elasticsearch
Key concepts of ElasticsearchIndexDocumentDocument typeMappingKey concepts of the Elasticsearch infrastructureNodes and clustersShardsReplicasGatewayIndexing and searching
Installing and configuring your cluster
Installing JavaInstalling ElasticsearchRunning ElasticsearchShutting down ElasticsearchThe directory layoutConfiguring ElasticsearchThe system-specific installation and configurationInstalling Elasticsearch on LinuxInstalling Elasticsearch using RPM packagesInstalling Elasticsearch using the DEB packageElasticsearch configuration file localizationConfiguring Elasticsearch as a system service on LinuxElasticsearch as a system service on Windows
Manipulating data with the REST API
Understanding the REST APIStoring data in ElasticsearchCreating a new documentAutomatic identifier creationRetrieving documentsUpdating documentsDealing with non-existing documentsAdding partial documentsDeleting documentsVersioningUsage exampleVersioning from external systems
Searching with the URI request query
Sample dataURI searchElasticsearch query responseQuery analysisURI query string parametersThe queryThe default search fieldAnalyzerThe default operator propertyQuery explanationThe fields returnedSorting the resultsThe search timeoutThe results windowLimiting per-shard resultsIgnoring unavailable indicesThe search typeLowercasing term expansionWildcard and prefix analysisLucene query syntax
Summary
2. Indexing Your Data
Elasticsearch indexingShards and replicasWrite consistencyCreating indicesAltering automatic index creationSettings for a newly created indexIndex deletion
Mappings configuration
Type determining mechanismDisabling the type determining mechanismTuning the type determining mechanism for numeric typesTuning the type determining mechanism for datesIndex structure mappingType and types definitionFieldsCore typesCommon attributesStringNumberBooleanBinaryDateMulti fieldsThe IP address typeToken count typeUsing analyzersOut-of-the-box analyzersDefining your own analyzersDefault analyzersDifferent similarity modelsSetting per-field similarityAvailable similarity modelsConfiguring default similarityConfiguring BM25 similarityConfiguring DFR similarityConfiguring IB similarity
Batch indexing to speed up your indexing process
Preparing data for bulk indexingIndexing the dataThe _all fieldThe _source fieldAdditional internal fields
Introduction to segment merging
Segment mergingThe need for segment mergingThe merge policyThe merge schedulerThrottling
Introduction to routing
Default indexingDefault searchingRoutingThe routing parametersRouting fields
Summary
3. Searching Your Data
Querying ElasticsearchThe example dataA simple queryPaging and result sizeReturning the version valueLimiting the scoreChoosing the fields that we want to returnSource filteringUsing the script fieldsPassing parameters to the script fields
Understanding the querying process
Query logicSearch typeSearch execution preferenceSearch shards API
Basic queries
The term queryThe terms queryThe match all queryThe type queryThe exists queryThe missing queryThe common terms queryThe match queryThe Boolean match queryThe phrase match queryThe match phrase prefix queryThe multi match queryThe query string queryRunning the query string query against multiple fieldsThe simple query string queryThe identifiers queryThe prefix queryThe fuzzy queryThe wildcard queryThe range queryRegular expression queryThe more like this query
Compound queries
The bool queryThe dis_max queryThe boosting queryThe constant_score queryThe indices query
Using span queries
A spanSpan term querySpan first querySpan near querySpan or querySpan not querySpan within querySpan containing querySpan multi queryPerformance considerations
Choosing the right query
The use casesLimiting results to given tagsSearching for values in a rangeBoosting some of the matched documentsIgnoring lower scoring partial queriesUsing Lucene query syntax in queriesHandling user queries without errorsAutocomplete using prefixesFinding terms similar to a given oneMatching phrasesSpans, spans everywhere
Summary
4. Extending Your Querying Knowledge
Filtering your resultsThe context is the keyExplicit filtering with bool query
Highlighting
Getting started with highlightingField configurationUnder the hoodForcing highlighter typeConfiguring HTML tagsControlling highlighted fragmentsGlobal and local settingsRequire matchingCustom highlighting queryThe Postings highlighter
Validating your queries
Using the Validate API
Sorting data
Default sortingSelecting fields used for sortingSorting modeSpecifying behavior for missing fieldsDynamic criteriaCalculate scoring when sorting
Query rewrite
Prefix query as an exampleGetting back to Apache LuceneQuery rewrite properties
Summary
5. Extending Your Index Structure
Indexing tree-like structuresData structureAnalysis
Indexing data that is not flat
DataObjectsArraysMappingsFinal mappingsSending the mappings to ElasticsearchTo be or not to be dynamicDisabling object indexing
Using nested objects
Scoring and nested queries
Using the parent-child relationship
Index structure and data indexingChild mappingsParent mappingsThe parent documentChild documentsQueryingQuerying data in the child documentsQuerying data in the parent documentsPerformance considerations
Modifying your index structure with the update API
The mappingsAdding a new field to the existing indexModifying fields of an existing index
Summary
6. Make Your Search Better
Introduction to Apache Lucene scoringWhen a document is matchedDefault scoring formulaRelevancy matters
Scripting capabilities of Elasticsearch
Objects available during script executionScript typesIn file scriptsInline scriptsIndexed scriptsQuerying with scriptsScripting with parametersScript languagesUsing other than embedded languagesUsing native codeThe factory implementationImplementing the native scriptThe plugin definitionInstalling the pluginRunning the script
Searching content in different languages
Handling languages differentlyHandling multiple languagesDetecting the language of the documentSample documentThe mappingsQueryingQueries with an identified languageQueries with an unknown languageCombining queries
Influencing scores with query boosts
The boostAdding the boost to queriesModifying the scoreConstant score queryBoosting queryThe function score queryStructure of the function queryThe weight factor functionField value factor functionThe script score functionThe random score functionDecay functions
When does index-time boosting make sense?
Defining boosting in the mappings
Words with the same meaning
Synonym filterSynonyms in the mappingsSynonyms stored on the file systemDefining synonym rulesUsing Apache Solr synonymsExplicit synonymsEquivalent synonymsExpanding synonymsUsing WordNet synonymsQuery or index-time synonym expansion
Understanding the explain information
Understanding field analysisExplaining the query
Summary
7. Aggregations for Data Analysis
AggregationsGeneral query structureInside the aggregations engine
Aggregation types
Metrics aggregationsMinimum, maximum, average, and sumMissing valuesUsing scriptsField value statistics and extended statisticsValue countField cardinalityPercentilesPercentile ranksTop hits aggregationAdditional parametersGeo bounds aggregationScripted metrics aggregationBuckets aggregationsFilter aggregationFilters aggregationTerms aggregationCounts are approximateMinimum document countRange aggregationKeyed bucketsDate range aggregationIPv4 range aggregationMissing aggregationHistogram aggregationDate histogram aggregationTime zonesGeo distance aggregationsGeohash grid aggregationGlobal aggregationSignificant terms aggregationChoosing significant termsMultiple value analysisSampler aggregationChildren aggregationNested aggregationReverse nested aggregationNesting aggregations and ordering bucketsBuckets ordering
Pipeline aggregations
Available typesReferencing other aggregationsGaps in the dataPipeline aggregation typesMin, max, sum, and average bucket aggregationsCumulative sum aggregationBucket selector aggregationBucket script aggregationSerial differencing aggregationDerivative aggregationMoving avg aggregationPredicting future bucketsThe models
Summary
8. Beyond Full-text Searching
PercolatorThe indexPercolator preparationGetting deeperControlling the size of returned resultsPercolator and score calculationCombining percolators with other functionalitiesGetting the number of matching queriesIndexed document percolation
Elasticsearch spatial capabilities
Mapping preparation for spatial searchesExample dataAdditional geo_field propertiesSample queriesDistance-based sortingBounding box filteringLimiting the distanceArbitrary geo shapesPointEnvelopePolygonMultipolygonAn example usageStoring shapes in the index
Using suggesters
Available suggester typesIncluding suggestionsSuggester responseTerm suggesterTerm suggester configuration optionsAdditional term suggester optionsPhrase suggesterConfigurationCompletion suggesterIndexing dataQuerying indexed completion suggester dataCustom weightsContext suggesterContext typesUsing contextUsing the geo location context
The Scroll API
Problem definitionScrolling to the rescue
Summary
9. Elasticsearch Cluster in Detail
Understanding node discoveryDiscovery typesNode rolesMaster nodeData nodeClient nodeConfiguring node rolesSetting the cluster's nameZen discoveryMaster election configurationConfiguring unicastFault detection ping settingsCluster state updates controlDealing with master unavailabilityAdjusting HTTP transport settingsDisabling HTTPHTTP portHTTP host
The gateway and recovery modules
The gatewayRecovery controlAdditional gateway recovery optionsIndices recovery APIDelayed allocationIndex recovery prioritization
Templates and dynamic templates
TemplatesAn example of a templateDynamic templatesThe matching patternField definitions
Elasticsearch plugins
The basicsInstalling pluginsRemoving plugins
Elasticsearch caches
Fielddata cacheFielddata sizeCircuit breakersFielddata and doc valuesShard request cacheEnabling and configuring the shard request cachePer request shard request cache disablingShard request cache usage monitoringNode query cacheIndexing buffersWhen caches should be avoided
The update settings API
The cluster settings APIThe indices settings API
Summary
10. Administrating Your Cluster
Elasticsearch time machineCreating a snapshot repositoryCreating snapshotsAdditional parametersRestoring a snapshotCleaning up – deleting old snapshots
Monitoring your cluster's state and health
Cluster health APIControlling information detailsAdditional parametersIndices stats APIDocsStoreIndexing, get, and searchAdditional informationNodes info APIReturned informationNodes stats APICluster state APICluster stats APIPending tasks APIIndices recovery APIIndices shard stores APIIndices segments API
Controlling the shard and replica allocation
Explicitly controlling allocationSpecifying node parametersConfigurationIndex creationExcluding nodes from allocationRequiring node attributesUsing the IP address for shard allocationDisk-based shard allocationConfiguring disk based shard allocationDisabling disk based shard allocationThe number of shards and replicas per nodeAllocation throttlingCluster-wide allocationAllocation awarenessForcing allocation awarenessFilteringWhat do include, exclude, and require meanManually moving shards and replicasMoving shardsCanceling shard allocationForcing shard allocationMultiple commands per HTTP requestAllowing operations on primary shardsHandling rolling restarts
Controlling cluster rebalancing
Understanding rebalanceCluster being readyThe cluster rebalance settingsControlling when rebalancing will be allowedControlling the number of shards being moved between nodes concurrentlyControlling which shards may be rebalanced
The Cat API
The basicsUsing Cat APICommon argumentsThe examplesGetting information about the master nodeGetting information about the nodesRetrieving recovery information for an index
Warming up
Defining a new warming queryRetrieving the defined warming queriesDeleting a warming queryDisabling the warming up functionalityChoosing queries for warming
Index aliasing and using it to simplify your everyday work
An aliasCreating an aliasModifying aliasesCombining commandsRetrieving aliasesRemoving aliasesFiltering aliasesAliases and routingZero downtime reindexing and aliases
Summary
11. Scaling by Example
HardwarePhysical servers or a cloudCPURAM memoryMass storageThe networkHow many serversCost cutting
Preparing a single Elasticsearch node
The general preparationsAvoiding swappingFile descriptorsVirtual memoryThe memoryField data cache and breaking the circuitUse doc valuesRAM buffer for indexingIndex refresh rateThread pools
Horizontal expansion
Automatically creating the replicasRedundancy and high availabilityCost and performance flexibilityContinuous upgradesMultiple Elasticsearch instances on a single physical machinePreventing a shard and its replicas from being on the same nodeDesignated node roles for larger clustersQuery aggregator nodesData nodesMaster eligible nodes
Preparing the cluster for high indexing and querying throughput
Indexing related adviceIndex refresh rateThread pools tuningAutomatic store throttlingHandling time-based dataMultiple data pathsData distributionBulk indexingRAM buffer for indexingAdvice for high query rate scenariosShard request cacheThink about the queriesParallelize your queriesField data cache and breaking the circuitKeep size and shard size under control
Monitoring
Elasticsearch HQMarvelSPM for Elasticsearch
Summary
Index

Content preview from Elasticsearch Server - Third Edition - Third Edition

Installing and configuring your cluster

Installing and running Elasticsearch even in production environments is very easy nowadays, compared to how it was in the days of Elasticsearch 0.20.x. From a system that is not ready to one with Elasticsearch, there are only a few steps that one needs to go. We will explore these steps in the following section:

Installing Java

Elasticsearch is a Java application and to use it we need to make sure that the Java SE environment is installed properly. Elasticsearch requires Java Version 7 or later to run. You can download it from http://www.oracle.com/technetwork/java/javase/downloads/index.html. You can also use OpenJDK (http://openjdk.java.net/) if you wish. You can, of course, use Java Version 7, but it is ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Mastering Elasticsearch 5.x - Third Edition

Publisher Resources

ISBN: 9781785888816

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Elasticsearch Server - Third Edition - Third Edition

by Marek Rogozinski, Rafal Kuc

Installing and configuring your cluster

Installing Java

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.