Overfitting on the training data is one of the biggest challenges of machine learning. There are many machine learning algorithms that are able to train on the training data by remembering all cases. In this scenario, the algorithm might not be able to generalize and make a correct prediction on new data. This is an especially big threat for deep learning, where neural networks have large numbers of trainable parameters. Therefore, it is extremely important to create a representative validation set.
In deep learning, the general advice when tackling new problems is to overfit as much as you can on the training data first. This ensures that your model is able to train on the training data and is ...
