Python Deep Learning Cookbook

How to do it...

1. We start by importing all the libraries as follows:

from nltk.tokenize import word_tokenize from nltk.stem import WordNetLemmatizer import numpy as np import random import pickle from collections import Counter import tensorflow as tf

2. Next, we process the English sentences with the nltk package. We start by defining the functions we need for preprocessing:

 

lemmatizer = WordNetLemmatizer() pos = '../Data/positive.txt'neg = '../Data/negative.txt'def create_lexicon(pos, neg):     lexicon = []     for fi in [pos, neg]:         with open(fi, 'r') as f:             contents = f.readlines()             for l in contents[:10000000]:                 all_words = word_tokenize(l.lower())                 lexicon += list(all_words)     lexicon = [lemmatizer.lemmatize(i) for i in lexicon]  w_counts ...