Splitting data

The objective of machine learning is to develop an algorithm that performs well on future data, which, by definition, is unseen. How can we simulate the behavior of feature data? We split a dataset into training and test data. Some of the dataset is held back, and we develop our algorithm using the training data.

The first technique is to split data into a training dataset and a testing dataset. We use the training data to develop our algorithm. We then see how well the algorithm generalizes by applying the trained algorithm to the test data and quantifying the error rate.

The train_test_split() function, from scikit-learn (sklearn), makes splitting data easy, as seen in the following steps:

  1. We will import this function, and ...

Get Training Systems Using Python Statistical Modeling now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.