O'Reilly logo

Effective Amazon Machine Learning by Alexis Perrier

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Splitting the dataset with shell commands

The command line is an often forgotten but powerful ally to the data scientist. Many very powerful operations on the data can be achieved with the right shell commands and executed blazingly fast. To illustrate this, we will use shell commands to shuffle, split, and create training and validation subsets of the Ames Housing dataset:

  1. First, extract the first line into a separate file, ames_housing_header.csv and remove it from the original file:
        $ head -n 1 ames_housing.csv > ames_housing_header.csv
  1. We just tail all the lines after the first one into the same file:
        $ tail -n +2 ames_housing.csv > ames_housing_nohead.csv
  1. Then randomly sort the rows into a temporary file. (gshuf is the OSX ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required