To summarize, Scrapy is designed to have the downloader as a bottleneck. Start with a low value of
CONCURRENT_REQUESTS and increase until just before you hit one of the following limits:
Responses in your scraper
At the same time also perform the following:
Requests at all times in the scheduler's queues (mqs/dqs) to prevent the downloader's URL starvation
Figure 13 summarizes the procedure of diagnosing and repairing Scrapy's ...