Deep Learning Quick Reference

Preparing the data

When working with text documents like this it can take a lot of mundane code to get you where you want to be. I'm including this example as a way to handle the problem. Once you understand what's going on here, you will be able to reuse much of it in future problems and shorten your development time, so it's worth the consideration.

The following function is going to take the top-level directory where the 20 newsgroup texts live. Within that directory, there will be 20 individual directories, each with files. Each file is a newsgroup post:

def load_data(text_data_dir, vocab_size, sequence_length, validation_split=0.2):    data = dict()    data["vocab_size"] = vocab_size    data["sequence_length"] = sequence_length    # second, prepare ...