Python Web Scraping Cookbook

Handling basic authorization

Some websites use a form of authorization known as basic authorization. This was popular before other means of authorization, such as cookie auth or OAuth. It is also common on corporate intranets and some web APIs. In basic authorization, a header is added to the HTTP request. This header, Authorization, is passed the Basic string and then a base64 encoding of the values <username>:<password>. So in the case of darkhelmet, this header would look as follows:

Authorization: Basic ZGFya2hlbG1ldDp2ZXNwYQ==, with ZGFya2hlbG1ldDp2ZXNwYQ== being darkhelmet:vespa base 64 encoded.

Note that this is no more secure than sending it in plain-text, (although when performed over HTTPS it is secure.) However, for the most part, ...