Learning Data Mining with Python - Second Edition

Extracting the content

After we get the raw data, we need to find the story in each. There are several complex algorithms for doing this, as well as some simple ones. We will stick with a simple method here, keeping in mind that often enough, the simple algorithm is good enough. This is part of data mining—knowing when to use simple algorithms to get a job done, versus using more complicated algorithms to obtain that extra bit of performance.

First, we get a list of each of the filenames in our raw subfolder:

filenames = [os.path.join(data_folder, filename) for filename in os.listdir(data_folder)]

Next, we create an output folder for the text-only versions that we will extract:

 text_output_folder = os.path.join(os.path.expanduser("~"), ...