Mastering TensorFlow 1.x

Loading and preparing the PTB dataset

Start by importing the modules and loading the data as follows::

from datasetslib.ptb import PTBSimpleptb = PTBSimple()# downloads data, converts words to ids, converts files to a list of idsptb.load_data() print('Train :',ptb.part['train'][0:5])print('Test: ',ptb.part['test'][0:5])print('Valid: ',ptb.part['valid'][0:5])print('Vocabulary Length = ',ptb.vocab_len)

The first five elements of each of the datasets along with the vocabulary length are printed as follows:

Train : [9970, 9971, 9972, 9974, 9975]
Test:  [102, 14, 24, 32, 752]
Valid:  [1132, 93, 358, 5, 329]
Vocabulary Length =  10000

We set the window of context to two words and get the CBOW pairs:

ptb.skip_window=2ptb.reset_index_in_epoch()# in ...