Python Web Scraping - Second Edition

Expiring stale data

Our current version of the disk cache will save a value to disk for a key and then return it whenever this key is requested in the future. This functionality may not be ideal when caching web pages because of online content changes, so the data in our cache will become out of date. In this section, we will add an expiration time to our cached data so the crawler knows when to download a fresh copy of the web page. To support storing the timestamp of when each web page was cached is straightforward.

Here is an implementation of this:

from datetime import datetime, timedelta class DiskCache:     def __init__(..., expires=timedelta(days=30)):         ...         self.expires = expires

## in __getitem___ for DiskCache classwith open(path, ...