O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 6. Deploying to Scrapinghub

In the last few chapters, we took a look at how to develop Scrapy spiders. As soon as we're satisfied with their functionality, we have two options. If all we want is to use them for a single scrape, we may be okay with letting them run for some time on our dev machine. On the other hand, quite often, we need to run scraping jobs periodically. We may use cloud servers from Amazon, RackSpace, or any other provider, but this requires some setup, configuration and maintenance. This is where Scrapinghub comes into play.

Scapinghub is the Amazon of Scrapy hosting—a cloud Scrapy infrastructure provider that is built by lead Scrapy developers. It is a paid service, but they provide a free tier with no need for a credit ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required