Hands-On Data Analysis with Pandas

Training and testing sets

So far, machine learning sounds pretty great, though—we can build a model that will learn how to perform a task for us. Therefore, we should give it all the data we have so that it learns well, right? Unfortunately, it's not that simple. If we give the model all of our data, we risk overfitting it, meaning that it won't be able to generalize well to new data points because it was fit to the sample rather than the population. On the other hand, if we don't give it enough data, it will underfit and be unable to capture the underlying information in the data.

When a model fits the randomness in the data, it is said to fit the noise in the data.

Another thing to consider is that, if we use all of our data to train the ...