Technical requirementsLexiconsPhonemes, graphemes, and morphemesTokenizationIssues with tokenizationDifferent types of tokenizersRegular expressionsRegular expressions-based tokenizersTreebank tokenizerTweetTokenizerUnderstanding word normalizationStemmingOver-stemming and under-stemmingLemmatizationWordNet lemmatizerSpacy lemmatizerStopword removalCase foldingN-gramsTaking care of HTML tagsHow does all this fit into my NLP pipeline?Summary