Modern Python Standard Library Cookbook

Memory and disk buffer

Sometimes, we need to keep certain data in a buffer, such as a file we downloaded from the internet or some data we are generating on the fly.

As the size of such data is not always predictable, is usually not a good idea to keep it all in memory.

If you are downloading a big 32 GB file from the internet that you need to process (such as decompress or parse), it will probably exhaust all your memory if you try to store it into a string before processing it.

That's why it's usually a good idea to rely on tempfile.SpooledTemporaryFile, which will keep the content in memory until it reaches its maximum size and will then move it to a temporary file if it's bigger than the maximum allowed size.

That way, we can have the ...