O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Overview of our distributed system

Designing this system was a great experience for me. I started adding features and complexity to the point where I had to demand that readers have high-end hardware to run the examples. What then became an urgent necessity was simplicity—both in order to keep the hardware requirements realistic and to ensure that this chapter remains focused on Scrapy.

Overview of our distributed system

Overview of the system

At the end, the system that we are going to use in this chapter will contain our dev machine and a few more servers. We will use our dev machine to perform the horizontal crawling of the index and extract batches of URLs. We will then distribute ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required