Webbots, Spiders, and Screen Scrapers, 2nd Edition

Making Data Smaller

Now that you know how to store data, you’ll want to efficiently store the data in ways that reduce the amount of disk spaced required, while facilitating easy retrieval and manipulation of that data. The following section explores methods for reducing the size of the data your webbots collect in these ways:

• Storing references to data

• Compressing data

• Removing unneeded formatting

• Thumbnailing or creating smaller representations of larger graphic files

Storing References to Image Files

Since your webbot and the image files it discovers share the same network, it is possible to store a network reference to the image instead of making a physical copy of it. For example, instead of downloading and storing the image north_beach.jpg from ...