Hands-On Machine Learning for Algorithmic Trading

Extracting data from HTML using requests and BeautifulSoup

In this section, we will request and parse HTML source code. We will be using the requests library to make Hyper Text Transfer Protocol (HTTP) requests and retrieve the HTML source code, and BeautifulSoup to parse and extract the text content.

We will, however, encounter a common obstacle: websites may request certain information from the server only after initial page-load using JavaScript. As a result, a direct HTTP request will not be successful. To sidestep this type of protection, we will use a headless browser that retrieves the website content as a browser would:

from bs4 import BeautifulSoupimport requests# set and request url; extract source codeurl = "https://www.opentable.com/new-york-restaurant-listings" ...