Python Web Scraping - Second Edition

Selenium

With the WebKit library used in the previous section, we have full control to customize the browser renderer to behave as we need it to. If this level of flexibility is not needed, a good and easier-to-install alternative is Selenium, which provides an API to automate several popular web browsers. Selenium can be installed using pip with the following command:

pip install selenium

To demonstrate how Selenium works, we will rewrite the previous search example in Selenium. The first step is to create a connection to the web browser:

>>> from selenium import webdriver >>> driver = webdriver.Firefox() 

When this command is run, an empty browser window will pop up. If you received an error instead, you likely need to install geckodriver ...