O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Summary

In this chapter, we tried to give you some interesting cases that highlight the fine performance implications of Scrapy's architecture. Details might change in future versions of Scrapy, but the intuition provided by this chapter should remain valid for a long time and might also help you with any high-concurrency asynchronous systems that are based on Twisted, Netty Node.js, or similar frameworks.

When it comes to the question of performance in Scrapy, there are three valid answers: I don't know and I don't care, I don't know but I will find out, or I do know. As we demonstrated many times in this chapter, the naive answer, "we need more servers/memory/bandwidth" is most likely irrelevant to Scrapy's performance. One really needs to understand ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required