Mastering TensorFlow 1.x

Loading and preparing the text8 dataset

Now we do the same loading and preprocessing steps with the text8 dataset:

from datasetslib.text8 import Text8text8 = Text8()text8.load_data() # downloads data, converts words to ids, converts files to a list of idsprint('Train:', text8.part['train'][0:5])print('Vocabulary Length = ',text8.vocab_len)

We find the vocabulary length is about 254,000 words:

Train: [5233, 3083, 11, 5, 194]
Vocabulary Length =  253854

Some of the tutorials manipulate this data by finding the most common words or truncate the vocabulary size to 10,000 words. However, we have used the complete dataset and complete vocabulary from the first file of the text8 dataset.

Preparing the CBOW pairs:

text8.skip_window=2text8.reset_index_in_epoch() ...