Scraping the web with Beautiful Soup

In the previous section, we discussed how to operate on textual data, given the fact that we already have the dataset. What if we need to scrape the web and download it manually? This process happens more often than you can expect, and it's a very popular topic of interest in data science. For example:

  • Financial institutions scrape the web to extract fresh details and information about the companies in their portfolio. Newspapers, social networks, blogs, forums, and corporate websites are the ideal targets for these analyses.
  • Advertisement and media companies analyze sentiment and the popularity of many pieces of the web to understand people's reactions.
  • Companies specialized in insight analysis and recommendation ...

Get Python Data Science Essentials - Third Edition now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.