Twisted Network Programming Essentials

Chapter 3. Web Clients

The most common way to interact with the Web is through a web browser. But as more data and services are made available on the Web, it’s important to be able to write web clients that can communicate with web servers through HTTP. This chapter shows how to use the twisted.web.client module to interact with web resources, including downloading pages, using HTTP authentication, uploading files, and working with HTTP headers.

Downloading a Web Page

The simplest and most common task for a web client application is fetching the contents of a web page. The client connects to the server, sends an HTTP GET request, and receives an HTTP response containing the requested page.

How Do I Do That?

Here’s where you can begin to experience the usefulness of Twisted’s built-in protocol support. The twisted.web package includes a complete HTTP implementation, saving you the work of developing the necessary Protocol and ClientFactory classes. Furthermore, it includes utility functions that allow you to make an HTTP request with a single function call. To fetch the contents of a web page, use the function twisted.web.client.getPage. Example 3-1 is a Python script called webcat.py, which fetches a URL that you specify.

Example 3-1. webcat.py

from twisted.web import client from twisted.internet import reactor import sys def printPage(data): print data reactor.stop() def printError(failure): print >> sys.stderr, "Error:", failure.getErrorMessage() reactor.stop() if len(sys.argv) == 2: ...