October 2019
Beginner
186 pages
5h 39m
English
The standard tokenizer uses Unicode Text Segmentation to divide the text. Consider the following example:
POST _analyze { "tokenizer": "standard", "text": "Those who dare to fail miserably can achieve greatly."}
This will output the following result:
[Those, who, dare, to, fail, miserably, can, achieve, greatly]
It accepts the max_token_length parameter, which will split the token if it exceeds the specified length. By default, this is set to 255.
Read now
Unlock full access