Python Cookbook

Resuming the HTTP Download of a File

Credit: Chris Moffitt

Problem

You need to resume an HTTP download of a file that has been partially transferred.

Solution

Large downloads are sometimes interrupted. However, a good HTTP server that supports the Range header lets you resume the download from where it was interrupted. The standard Python module urllib lets you access this functionality almost seamlessly. You need to add only the needed header and intercept the error code the server sends to confirm that it will respond with a partial file:

import urllib, os

class myURLOpener(urllib.FancyURLopener):
    """ Subclass to override error 206 (partial file being sent); okay for us """
    def http_error_206(self, url, fp, errcode, errmsg, headers, data=None):
        pass    # Ignore the expected "non-error" code

def getrest(dlFile, fromUrl, verbose=0):
    loop = 1
    existSize = 0
    myUrlclass = myURLOpener(  )
    if os.path.exists(dlFile):
        outputFile = open(dlFile,"ab")
        existSize = os.path.getsize(dlFile)
        # If the file exists, then download only the remainder
        myUrlclass.addheader("Range","bytes=%s-" % (existSize)) else: outputFile = open(dlFile,"wb") webPage = myUrlclass.open(fromUrl) if verbose: for k, v in webPage.headers.items( ): print k, "=", v # If we already have the whole file, there is no need to download it again numBytes = 0 webSize = int(webPage.headers['Content-Length']) if webSize == existSize: if verbose: print "File (%s) was already downloaded from URL (%s)"%( dlFile, fromUrl) else: if verbose: print ...