Learning Data Mining with Python - Second Edition

Finding the stories in arbitrary websites

To start with, we will download the full web page from each of these links and store them in our data folder, under a raw subfolder. We will process these to extract the useful information later on. This caching of results ensures that we don't have to continuously download the websites while we are working. First, we set up the data folder path:

import os data_folder = os.path.join(os.path.expanduser("~"), "Data", "websites", "raw")

We are going to use MD5 hashing to create unique filenames for our articles, by hashing the URL, and we will import hashlib to do that. A hash function is a function that converts some input (in our case a string containing the title) into a string that is seemingly ...