Hands-On Unsupervised Learning with Python

Sparse autoencoders

The code generated by a standard autoencoder is generally dense; however, as discussed in Chapter 7, Dimensionality Reduction and Component Analysis, sometimes, it's preferable to work with over-complete dictionaries and sparse encodings. The main strategy to accomplish this goal is to simply add an L₁ penalty (on the code layer) to the cost function:

The α constant determines the amount of sparseness that will be reached. Of course, as the optimum of C_s doesn't correspond to the original one, in order to achieve the same accuracy, more epochs and a longer code layer are often needed. Another method, proposed by Andrew ...