B SCRAPING THE WEB
Sometimes, in order to research important data publicly available online, you’ll need to download a local copy. When websites don’t provide this data in structured downloadable formats like spreadsheets, JSON files, or databases, you can make your own copy using web scraping (or screen scraping): writing code that loads web pages for you and extracts their contents. These might include social media posts, court documents, or any other online data. You can use web scraping to download either full datasets or the same web page again and again on a regular basis to see if its content changes over time.
For example, consider the Parler dataset discussed in Chapter 11. Before Parler was kicked offline by its hosting provider ...
Get Hacks, Leaks, and Revelations now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
