Machine Learning Algorithms - Second Edition

Fully connected layers

A fully connected layer (sometimes called a dense layer) is made up of n neurons, and each of them receives all the output values coming from the previous layer (such as the hidden layer in a Multi-layer Perceptron (MLP)). It can be characterized by a weight matrix, a bias vector, and an activation function:

It's important to remember that an MLP must contain non-linear activations (for example, sigmoid, hyperbolic tangent, or ReLU). In fact, a network with n linear hidden layers is equivalent to a standard perceptron. In complex architectures, they are normally used as intermediate or output layers, in particular when ...