The data reader script helps to generate trainable batches from the preprocessed training text from the data parser script. Let's start by importing the required methods:
import pickleimport random
This helper module helps generate trainable batches from the preprocessed training text.
class Data_Reader: def __init__(self, cur_train_index=0, load_list=False): self.training_data = pickle.load(open('data/conversations_lenmax22_formersents2_with_former', 'rb')) self.data_size = len(self.training_data) if load_list: self.shuffle_list = pickle.load(open('data/shuffle_index_list', 'rb')) else: self.shuffle_list = self.shuffle_index() self.train_index = cur_train_index
The following code gets the batch number from the data:
def get_batch_num ...