Natural Language Processing with Python

Summary

• Texts are represented in Python using lists: ['Monty', 'Python']. We can use indexing, slicing, and the len() function on lists.

• A word “token” is a particular appearance of a given word in a text; a word “type” is the unique form of the word as a particular sequence of letters. We count word tokens using len(text) and word types using len(set(text)).

• We obtain the vocabulary of a text t using sorted(set(t)).

• We operate on each item of a text using [f(x) for x in text].

• To derive the vocabulary, collapsing case distinctions and ignoring punctuation, we can write set([w.lower() for w in text if w.isalpha()]).

• We process each word in a text using a for statement, such as for w in t: or for word in text:. This must be followed by the colon character and an indented block of code, to be executed each time through the loop.

• We test a condition using an if statement: if len(word) < 5:. This must be followed by the colon character and an indented block of code, to be executed only if the condition is true.

• A frequency distribution is a collection of items along with their frequency counts (e.g., the words of a text and their frequency of appearance).

• A function is a block of code that has been assigned a name and can be reused. Functions are defined using the def keyword, as in def mult(x, y); x and y are parameters of the function, and act as placeholders for actual data values.

• A function is called by specifying its name followed by one or more arguments inside parentheses, like this: ...