Machine Learning and AI with Simple Python and Matlab Scripts

13RNNs with Attention for Machine Translation

13.1 Introduction

In this chapter we introduce a simple example for machine translation using an RNN with the attention mechanism. As discussed in Chapter 12, for applications where it is important to remember the hidden states generated by previous words in translation, we need the attention mechanism.

In natural languages, placements of the subject, verb, preposition and tense information in a sentence significantly vary from one language to another. For example, if one would like to say The sky is viewed only by me in Polish, it would be Niebo oglądam tylko ja. However, the statement The sky is viewed only by you is translated into Polish as Niebo oglądasz tylko ty. As can be seen in Figures 13.1a and b, the translation of this sentence into Polish cannot be determined until the user enters the last English word either as me in the first sentence or you in the second. Entering the last word as me causes the inclusion of words oglądam and ja in the Polish translation, whereas the last word of you will place the words oglądasz and ty in the Polish sentence.

Another observation from Figure 13.1 is the difference between the number of words in the English input and Polish output sentences. As described in Section 11.4 in Chapter 11, one popular solution for handling input and output sequences with different lengths is adopting padding techniques, which insert blank characters for shorter sequences to make all inputs and outputs appear ...