Natural Language Processing with TensorFlow

Training an NMT jointly with word embeddings

Here we will discuss how we can train an NMT jointly with word embeddings. We will be covering two concepts in this section:

• Training an NMT jointly with a word embedding layer
• Using pretrained embeddings instead of randomly initializing the embeddings layer

There are several multilingual word embedding repositories available:

• Facebook's fastText: https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md
• CMU multilingual embeddings: http://www.cs.cmu.edu/~afm/projects/multilingual_embeddings.html

From these, we will use the CMU embeddings (~200 MB) as it's much smaller compared with fastText (~5 GB). We first need to download the German (multilingual_embeddings.de) and English (multilingual_embeddings.en ...