Python Web Scraping - Second Edition

Compression

To make this cache feature complete compared with the original disk cache, we need to add one final feature: compression. This can be achieved in a similar way to the disk cache by serializing the data and then compressing it with zlib, as follows:

import zlib from bson.binary import Binary class RedisCache:    def __init__(..., compress=True):        ...        self.compress = compress    def __getitem__(self, url):         record = self.client.get(url)        if record:            if self.compress:                record = zlib.decompress(record)            return json.loads(record.decode(self.encoding))        else:             raise KeyError(url + ' does not exist')     def __setitem__(self, url, result):         data = bytes(json.dumps(result), self.encoding)        if self.compress:            data = zlib.compress(data) self.client.setex(url, ...