Hands-On Transfer Learning with Python

Selective attention

The controller output is used to figure out which location of memory to read from or write to. This is defined by a set of weights spread over all memory locations, which add up to 1. The weights are defined by the following two mechanisms. The idea is to give the controller several different modes of reading or writing to memory, corresponding to different data structures:

• Content based: Compares the key k output of the controller with all the memory locations using a similarity measure, say, cosine similarity (S), and then all the distances are normalized by softmax to get weights that add up to 1:

In this, β ≥ 1 is ...