Python Web Scraping Cookbook

How it works

Let us break down the explanation:

1. We start by importing the required items from Selenium:

from selenium import webdriverfrom selenium.webdriver.support import ui

2. Now we load the driver and the page:

driver = webdriver.PhantomJS()driver.get("http://the-internet.herokuapp.com/dynamic_loading/2")

3. With the page loaded, we can retrieve the button:

button = driver.find_element_by_xpath("//*/div[@id='start']/button")

4. And then we can click the button:

button.click()print("clicked")

5. Next we create a WebDriverWait object:

wait = ui.WebDriverWait(driver, 10)

6. With this object, we can request Selenium's UI wait for certain events. This also sets a maximum wait of 10 seconds. Now using this, we can wait until we meet a criterion; ...