Advanced Deep Learning with Python

Encoder

In this section, we'll implement the encoder, which is composed of several different subcomponents. Let's start with the main definition and then dive into more details:

class Encoder(torch.nn.Module):    def __init__(self, block: EncoderBlock, N: int):        super(Encoder, self).__init__()        self.blocks = clones(block, N)        self.norm = LayerNorm(block.size)    def forward(self, x, mask):        """Iterate over all blocks and normalize"""        for layer in self.blocks:            x = layer(x, mask)        return self.norm(x)

It is fairly straightforward: the encoder is composed of self.blocks: N stacked instances of EncoderBlock, where each serves as input for the next. They are followed by LayerNorm normalization self.norm (we discussed these concepts in the The transformer ...