O'Reilly logo

Instant Web Scraping with Java by Ryan Mitchell

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Faster scraping with RMI (Advanced)

Now that we've learned how to speed up our code by using multiple threads on the same server, we will learn how to speed up our code by using multiple servers. Distributed computing is a powerful and tricky tool to master, but the basics are straightforward. If you are interested in RMI, I highly recommend that you pay close attention to the There's more... section at the end of this recipe.

Getting ready

Using multiple servers to scrape the Web can be useful for several reasons. For one, even with threading, the limitations of one machine can be a problem for large-scale applications. For another, while visiting a website repeatedly from a single IP address might be a big red flag, visiting a website repeatedly ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required