Learn Python by Building Data Science Applications

CSS and XPath selectors

As mentioned previously, Beautiful Soup parses the HTML from string to a Python object. Even parsed, this structure is not an easy thing to navigate. This is especially true for bulk retrieval when we operate on multiple pages at once due to the dynamic nature of the web. Even the same page can change constantly, with some elements being added or removed, let alone different pages, even those with apparently the same structure. This is the moment when you'll start appreciating well-defined and stable APIs!

To navigate HTML document structures, also known as Document Object Models (DOMs), two common and widely adopted techniques are used. The first one, CSS Selectors, is a pattern language built to work with HTML and ...