Python Web Scraping - Second Edition

Implementing DiskCache

In the previous section, we covered the limitations of file systems that need to be considered when building a disk-based cache, namely the restriction on which characters can be used, the filename length, and ensuring a file and directory are not created in the same location. Combining this code with logic to map a URL to a filename will form the main part of the disk cache. Here is an initial implementation of the DiskCache class:

import os import re from urllib.parse import urlsplit class DiskCache:     def __init__(self, cache_dir='cache', max_len=255):         self.cache_dir = cache_dir         self.max_len = max_len     def url_to_path(self, url):         """ Return file system path string for given URL"""  components = urlsplit(url)  ...