Deep Learning with PyTorch

torchtext.data

The torchtext.data instance defines a class called Field, which helps us to define how the data has to be read and tokenized. Let's look at the following example, which we will use for preparing our IMDB dataset:

from torchtext import dataTEXT = data.Field(lower=True, batch_first=True,fix_length=20)LABEL = data.Field(sequential=False)

In the preceding code, we define two Field objects, one for actual text and another for the label data. For actual text, we expect torchtext to lowercase all the text, tokenize the text, and trim it to a maximum length of 20. If we are building the application for a production environment, we may fix the length to a much larger number. But, for the toy example it works well. The Field constructor ...