June 2020
Intermediate to advanced
382 pages
11h 39m
English
Content preview from 40 Algorithms Every Programmer Should Know
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
Start your free trial
Tokenization
When we are working with NLP, the first job is to divide the text into a list of tokens. This process is called tokenization. The granularity of the resulting tokens will vary based on the objective—for example, each token can consist of the following:
A word
A combination of words
A sentence
A paragraph