O'Reilly logo

Instant Web Scraping with Java by Ryan Mitchell

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Submitting a basic form (Advanced)

A web scraper wouldn't be much good if it got stymied the first time it encountered a login form. In this recipe, we will learn how to submit forms to get information on the other side of the form, log in to a website, or simply send information to a web server in an automated way.

Getting ready

You probably have at least some familiarity with the two basic methods of web form submission—GET and POST. Submitting forms with GET is easy enough. Simply use your scraper to connect to a page at the following link:

http://example.com/thanksforsubmitting.html?field1=foo&field2=bar

You can even obtain Google results (after setting your headers appropriately, of course) by scraping the page at:

http://www.google.com/search?q=java ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required