Machine Learning with Swift

Nonlinearity layers

This are all kind of nonlinearities, that we’ve already discussed in the previous chapter: tanh, sigmoid, ReLU, and so on. You usually want to put them after a convolutional or a fully-connected layer.

Softmax is a generalization of a logistic function to vectors: while the logistic function squashes scalar values to be between 0 and 1, softmax squashes vectors so that its elements adds up to 1. In the statistics, probability of outcomes in discrete random distribution adds up to 1, so this function is really useful for the classification, where target variable is discrete.