Practical Predictive Analytics

Reading the data

In the last chapter, we saved the out_sd to an external parquet file. In the real world, you will be faced with analyzing multiple data sources. Often, these data sources will have similar schemas but will differ by the time period that they were written.

For example, log files can be archived in different directories, categorized by date, and you as an analyst will need to read in multiple files and concatenate them together.

So although each individual file may be small, when aggregated together, they will yield a much larger file.

Let's pretend that we will be performing some machine learning on several files that have been written over the past two days. For illustration purposes, we will use the same file for the two ...