Natural Language Processing with Python

A Closer Look at Python: Texts as Lists of Words

You’ve seen some important elements of the Python programming language. Let’s take a few moments to review them systematically.

Lists

What is a text? At one level, it is a sequence of symbols on a page such as this one. At another level, it is a sequence of chapters, made up of a sequence of sections, where each section is a sequence of paragraphs, and so on. However, for our purposes, we will think of a text as nothing more than a sequence of words and punctuation. Here’s how we represent text in Python, in this case the opening sentence of Moby Dick:

>>> sent1 = ['Call', 'me', 'Ishmael', '.']
>>>

After the prompt we’ve given a name we made up, sent1, followed by the equals sign, and then some quoted words, separated with commas, and surrounded with brackets. This bracketed material is known as a list in Python: it is how we store a text. We can inspect it by typing the name . We can ask for its length . We can even apply our own lexical_diversity() function to it .

>>> sent1 
['Call', 'me', 'Ishmael', '.']
>>> len(sent1)  4 >>> lexical_diversity(sent1) ...