Python Deep Learning Projects

Data pre-processing

Let's perform the data pre-processing to convert the raw data into its encoded form. We will extract fixed length sentences, encode them using a one-hot encoding process, and finally build a tensor of the (sequence, maxlen, unique_characters) shape, as shown in the following diagram. At the same time, we will prepare the target vector, y, to contain the associated next character that follows each extracted sequence.

The following is the code we'll use to pre-process the data:

# Length of extracted character sequencesmaxlen = 100# We sample a new sequence every 5 charactersstep = 5# List to hold extracted sequencessentences = []# List to hold the target characters next_chars = []# Extracting sentences and the next characters. ...