TensorFlow Machine Learning Projects

Pre-processing data

Once you have installed the requisite packages (can be found in a requirements.txt file with the code) to run this project and read the data, the next step is to preprocess the data:

1. The first step is to get the tokens/word list from the reviews. Remove any punctuation and make sure that all of the tokens are in lowercase:

def get_processed_tokens(text):'''Gets Token List from a Review'''filtered_text = re.sub(r'[^a-zA-Z0-9\s]', '', text) #Removing Punctuationsfiltered_text = filtered_text.split()filtered_text = [token.lower() for token in filtered_text]return filtered_text

For example, if we have an input This is a GREAT movie!!!!, our output should be this is a great movie.

2. Create a token_idx dictionary that maps ...