Python Web Scraping Cookbook

How to do it

Supporting basic auth in Scrapy is straightforward. To get this to work for a spider and a given site the spider is crawling, simply define the http_user, http_pass, and name fields in your scraper. The following demonstrates:

class SomeIntranetSiteSpider(CrawlSpider):
    http_user = 'someuser'
    http_pass = 'somepass'
    name = 'intranet.example.com'
    # .. rest of the spider code omitted ...