A web scraper wouldn't be much good if it got stymied the first time it encountered a login form. In this recipe, we will learn how to submit forms to get information on the other side of the form, log in to a website, or simply send information to a web server in an automated way.
You probably have at least some familiarity with the two basic methods of web form submission—GET and POST. Submitting forms with GET is easy enough. Simply use your scraper to connect to a page at the following link:
You can even obtain Google results (after setting your headers appropriately, of course) by scraping the page at: