Python Deep Learning Projects

Data prep for a language generation (RNN) decoder

The first step is to pre-process the captions.

We will perform a few basic pre-processing steps on the captions, such as the following:

• First, we'll tokenize the captions (for example, by splitting on spaces). This will help us to build a vocabulary of all the unique words in the data (for example, "playing", "football", and so on).
• Next, we'll limit the vocabulary size to the top 5,000 words to save memory. We'll replace all other words with the token unk (for unknown). You can obviously optimize that according to the use case.
• Finally, we will create a word --> index mapping and vice versa.
• We will then pad all sequences to be the same length as the longest one.

Here is the code for that: ...