In the previous chapter, we explored the programming techniques that we use to write Scrapy middlewares. In this chapter, we will focus on writing correct and efficient pipelines by showcasing various common use cases, including consuming REST APIs, interfacing with databases, performing CPU-intensive tasks, and interfacing with legacy services.
For this chapter, we will use several new servers that you can see on the right-hand side of the following diagram:
Vagrant should have already set them up for us, and we should be able to ping them from dev using their hostname, such as
ping es or