Hands-On Machine Learning for Algorithmic Trading

Parsing, tokenizing, and annotating a sentence

Parsed document content is iterable, and each element has numerous attributes produced by the processing pipeline. The following sample illustrates how to access the following attributes:

• .text: Original word text
• .lemma_: Word root
• .pos_: Basic POS tag
• .tag_: Detailed POS tag
• .dep_: Syntactic relationship or dependency between tokens
• .shape_: The shape of the word regarding capitalization, punctuation, or digits
• .is alpha: Check whether the token is alphanumeric
• .is stop: Check whether the token is on a list of common words for the given language

We iterate over each token and assign its attributes to a pd.DataFrame:

pd.DataFrame([[t.text, t.lemma_, t.pos_, t.tag_, t.dep_, t.shape_, t.is_alpha, ...