book

Elasticsearch: The Definitive Guide

by Clinton Gormley, Zachary Tong

January 2015

Intermediate to advanced

724 pages

13h 21m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Who Should Read This BookWhy We Wrote This BookElasticsearch VersionHow to Read This BookNavigating This BookOnline ResourcesConventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
Installing ElasticsearchInstalling MarvelRunning ElasticsearchViewing Marvel and SenseTalking to ElasticsearchJava APIRESTful API with JSON over HTTPDocument OrientedJSONFinding Your FeetLet’s Build an Employee DirectoryIndexing Employee DocumentsRetrieving a DocumentSearch LiteSearch with Query DSLMore-Complicated SearchesFull-Text SearchPhrase SearchHighlighting Our SearchesAnalyticsTutorial ConclusionDistributed NatureNext Steps
An Empty ClusterCluster HealthAdd an IndexAdd FailoverScale HorizontallyThen Scale Some MoreCoping with Failure
What Is a Document?Document Metadata_index_type_idOther MetadataIndexing a DocumentUsing Our Own IDAutogenerating IDsRetrieving a DocumentRetrieving Part of a DocumentChecking Whether a Document ExistsUpdating a Whole DocumentCreating a New DocumentDeleting a DocumentDealing with ConflictsOptimistic Concurrency ControlUsing Versions from an External SystemPartial Updates to DocumentsUsing Scripts to Make Partial UpdatesUpdating a Document That May Not Yet ExistUpdates and ConflictsRetrieving Multiple DocumentsCheaper in BulkDon’t Repeat YourselfHow Big Is Too Big?
Routing a Document to a ShardHow Primary and Replica Shards InteractCreating, Indexing, and Deleting a DocumentRetrieving a DocumentPartial Updates to a DocumentMultidocument PatternsWhy the Funny Format?
The Empty SearchhitstookshardstimeoutMulti-index, MultitypePaginationSearch LiteThe _all FieldMore Complicated Queries
Exact Values Versus Full TextInverted IndexAnalysis and AnalyzersBuilt-in AnalyzersWhen Analyzers Are UsedTesting AnalyzersSpecifying AnalyzersMappingCore Simple Field TypesViewing the MappingCustomizing Field MappingsUpdating a MappingTesting the MappingComplex Core Field TypesMultivalue FieldsEmpty FieldsMultilevel ObjectsMapping for Inner ObjectsHow Inner Objects are IndexedArrays of Inner Objects
Empty SearchQuery DSLStructure of a Query ClauseCombining Multiple ClausesQueries and FiltersPerformance DifferencesWhen to Use WhichMost Important Queries and Filtersterm Filterterms Filterrange Filterexists and missing Filtersbool Filtermatch_all Querymatch Querymulti_match Querybool QueryCombining Queries with FiltersFiltering a QueryJust a FilterA Query as a FilterValidating QueriesUnderstanding ErrorsUnderstanding Queries

SortingSorting by Field ValuesMultilevel SortingSorting on Multivalue FieldsString Sorting and MultifieldsWhat Is Relevance?Understanding the ScoreUnderstanding Why a Document MatchedFielddata
Query PhaseFetch PhaseSearch Optionspreferencetimeoutroutingsearch_typescan and scroll
Creating an IndexDeleting an IndexIndex SettingsConfiguring AnalyzersCustom AnalyzersCreating a Custom AnalyzerTypes and MappingsHow Lucene Sees DocumentsHow Types Are ImplementedAvoiding Type GotchasThe Root ObjectPropertiesMetadata: _source FieldMetadata: _all FieldMetadata: Document IdentityDynamic MappingCustomizing Dynamic Mappingdate_detectiondynamic_templatesDefault MappingReindexing Your DataIndex Aliases and Zero Downtime
Making Text SearchableImmutabilityDynamically Updatable IndicesDeletes and UpdatesNear Real-Time Searchrefresh APIMaking Changes Persistentflush APISegment Mergingoptimize API
Finding Exact Valuesterm Filter with Numbersterm Filter with TextInternal Filter OperationCombining FiltersBool FilterNesting Boolean FiltersFinding Multiple Exact ValuesContains, but Does Not EqualEquals ExactlyRangesRanges on DatesRanges on StringsDealing with Null Valuesexists Filtermissing Filterexists/missing on ObjectsAll About CachingIndependent Filter CachingControlling CachingFilter Order
Term-Based Versus Full-TextThe match QueryIndex Some DataA Single-Word QueryMultiword QueriesImproving PrecisionControlling PrecisionCombining QueriesScore CalculationControlling PrecisionHow match Uses boolBoosting Query ClausesControlling AnalysisDefault AnalyzersConfiguring Analyzers in PracticeRelevance Is Broken!
Multiple Query StringsPrioritizing ClausesSingle Query StringKnow Your DataBest Fieldsdis_max QueryTuning Best Fields Queriestie_breakermulti_match QueryUsing Wildcards in Field NamesBoosting Individual FieldsMost FieldsMultifield MappingCross-fields Entity SearchA Naive ApproachProblems with the most_fields ApproachField-Centric QueriesProblem 1: Matching the Same Word in Multiple FieldsProblem 2: Trimming the Long TailProblem 3: Term FrequenciesSolutionCustom _all Fieldscross-fields QueriesPer-Field BoostingExact-Value Fields
Phrase MatchingTerm PositionsWhat Is a PhraseMixing It UpMultivalue FieldsCloser Is BetterProximity for RelevanceImproving PerformanceRescoring ResultsFinding Associated WordsProducing ShinglesMultifieldsSearching for ShinglesPerformance
Postcodes and Structured Dataprefix Querywildcard and regexp QueriesQuery-Time Search-as-You-TypeIndex-Time OptimizationsNgrams for Partial MatchingIndex-Time Search-as-You-TypePreparing the IndexQuerying the FieldEdge n-grams and PostcodesNgrams for Compound Words
Theory Behind Relevance ScoringBoolean ModelTerm Frequency/Inverse Document Frequency (TF/IDF)Vector Space ModelLucene’s Practical Scoring FunctionQuery Normalization FactorQuery CoordinationIndex-Time Field-Level BoostingQuery-Time BoostingBoosting an Indext.getBoost()Manipulating Relevance with Query StructureNot Quite Notboosting QueryIgnoring TF/IDFconstant_score Queryfunction_score QueryBoosting by Popularitymodifierfactorboost_modemax_boostBoosting Filtered Subsetsfilter Versus queryfunctionsscore_modeRandom ScoringThe Closer, The BetterUnderstanding the price ClauseScoring with ScriptsPluggable Similarity AlgorithmsOkapi BM25Changing SimilaritiesConfiguring BM25Relevance Tuning Is the Last 10%
Using Language AnalyzersConfiguring Language AnalyzersPitfalls of Mixing LanguagesAt Index TimeAt Query TimeIdentifying LanguageOne Language per DocumentForeign WordsOne Language per FieldMixed-Language FieldsSplit into Separate FieldsAnalyze Multiple TimesUse n-grams
standard Analyzerstandard TokenizerInstalling the ICU Plug-inicu_tokenizerTidying Up Input TextTokenizing HTMLTidying Up Punctuation
In That CaseYou Have an AccentRetaining MeaningLiving in a Unicode WorldUnicode Case FoldingUnicode Character FoldingSorting and CollationsCase-Insensitive SortingDifferences Between LanguagesUnicode Collation AlgorithmUnicode SortingSpecifying a LanguageCustomizing Collations
Algorithmic StemmersUsing an Algorithmic StemmerDictionary StemmersHunspell StemmerInstalling a DictionaryPer-Language SettingsCreating a Hunspell Token FilterHunspell Dictionary FormatChoosing a StemmerStemmer PerformanceStemmer QualityStemmer DegreeMaking a ChoiceControlling StemmingPreventing StemmingCustomizing StemmingStemming in situIs Stemming in situ a Good Idea
Pros and Cons of StopwordsUsing StopwordsStopwords and the Standard AnalyzerMaintaining PositionsSpecifying StopwordsUsing the stop Token FilterUpdating StopwordsStopwords and Performanceand Operatorminimum_should_matchDivide and ConquerControlling PrecisionOnly High-Frequency TermsMore Control with Common TermsStopwords and Phrase QueriesPositions DataIndex OptionsStopwordscommon_grams Token FilterAt Index TimeUnigram QueriesBigram Phrase QueriesTwo-Word PhrasesStopwords and Relevance
Using SynonymsFormatting SynonymsExpand or contractSimple ExpansionSimple ContractionGenre ExpansionSynonyms and The Analysis ChainCase-Sensitive SynonymsMultiword Synonyms and Phrase QueriesUse Simple Contraction for Phrase QueriesSynonyms and the query_string QuerySymbol Synonyms
FuzzinessFuzzy QueryImproving PerformanceFuzzy match QueryScoring FuzzinessPhonetic Matching
BucketsMetricsCombining the Two
Adding a Metric to the MixBuckets Inside BucketsOne Final Modification
Returning Empty BucketsExtended ExampleThe Sky’s the Limit
Global Bucket
Filtered QueryFilter BucketPost FilterRecap
Intrinsic SortsSorting by a MetricSorting Based on “Deep” Metrics
Finding Distinct CountsUnderstanding the Trade-offsOptimizing for SpeedCalculating PercentilesPercentile MetricPercentile RanksUnderstanding the Trade-offs
significant_terms DemoRecommending Based on PopularityRecommending Based on Statistics
FielddataAggregations and AnalysisHigh-Cardinality Memory ImplicationsLimiting Memory UsageFielddata SizeMonitoring fielddataCircuit BreakerFielddata FilteringDoc ValuesEnabling Doc ValuesPreloading FielddataEagerly Loading FielddataGlobal OrdinalsIndex WarmersPreventing Combinatorial ExplosionsDepth-First Versus Breadth-First
Lat/Lon FormatsFiltering by Geo-Pointgeo_bounding_box FilterOptimizing Bounding Boxesgeo_distance FilterFaster Geo-Distance Calculationsgeo_distance_range FilterCaching geo-filtersReducing Memory UsageSorting by DistanceScoring by Distance
Mapping Geohashesgeohash_cell Filter
geo_distance Aggregationgeohash_grid Aggregationgeo_bounds Aggregation
Mapping geo-shapesprecisiondistance_error_pctIndexing geo-shapesQuerying geo-shapesQuerying with Indexed ShapesGeo-shape Filters and Caching
Application-side JoinsDenormalizing Your DataField CollapsingDenormalization and ConcurrencyRenaming Files and DirectoriesSolving Concurrency IssuesGlobal LockingDocument LockingTree Locking
Nested Object MappingQuerying a Nested ObjectSorting by Nested FieldsNested Aggregationsreverse_nested AggregationWhen to Use Nested Objects
Parent-Child MappingIndexing Parents and ChildrenFinding Parents by Their Childrenmin_children and max_childrenFinding Children by Their ParentsChildren AggregationGrandparents and GrandchildrenPractical ConsiderationsMemory UseGlobal Ordinals and LatencyMultigenerations and Concluding Thoughts
The Unit of ScaleShard OverallocationKagillion ShardsCapacity PlanningReplica ShardsBalancing Load with ReplicasMultiple IndicesTime-Based DataIndex per Time FrameIndex TemplatesRetiring DataMigrate Old IndicesOptimize IndicesClosing Old IndicesArchiving Old IndicesUser-Based DataShared IndexFaking Index per User with AliasesOne Big UserScale Is Not Infinite
Marvel for MonitoringCluster HealthDrilling Deeper: Finding Problematic IndicesBlocking for Status ChangesMonitoring Individual Nodesindices SectionOS and Process SectionsJVM SectionThreadpool SectionFS and Network SectionsCircuit BreakerCluster StatsIndex StatsPending Taskscat API
HardwareMemoryCPUsDisksNetworkGeneral ConsiderationsJava Virtual MachineTransport Client Versus Node ClientConfiguration ManagementImportant Configuration ChangesAssign NamesPathsMinimum Master NodesRecovery SettingsPrefer Unicast over MulticastDon’t Touch These Settings!Garbage CollectorThreadpoolsHeap: Sizing and SwappingGive Half Your Memory to LuceneDon’t Cross 32 GB!Swapping Is the Death of PerformanceFile Descriptors and MMapRevisit This List Before Production
Changing Settings DynamicallyLoggingSlowlogIndexing Performance TipsTest Performance ScientificallyUsing and Sizing Bulk RequestsStorageSegments and MergingOtherRolling RestartsBacking Up Your ClusterCreating the RepositorySnapshotting All Open IndicesSnapshotting Particular IndicesListing Information About SnapshotsDeleting SnapshotsMonitoring Snapshot ProgressCanceling a SnapshotRestoring from a SnapshotMonitoring Restore OperationsCanceling a RestoreClusters Are Living, Breathing Creatures

Overview

Whether you need full-text search or real-time analytics of structured data—or both—the Elasticsearch distributed search engine is an ideal way to put your data to work. This practical guide not only shows you how to search, analyze, and explore data with Elasticsearch, but also helps you deal with the complexities of human language, geolocation, and relationships.