Skip to Content
Data Wrangling with Python
book

Data Wrangling with Python

by Jacqueline Kazil, Katharine Jarmul
February 2016
Beginner to intermediate
508 pages
12h 27m
English
O'Reilly Media, Inc.
Content preview from Data Wrangling with Python

Chapter 12. Advanced Web Scraping: Screen Scrapers and Spiders

You’ve begun your web scraping skills development, learning how to decipher what, how, and where to scrape in Chapter 11. In this chapter, we’ll take a look at more advanced scrapers, like browser-based scrapers and spiders to gather content.

We’ll also learn about debugging common problems with advanced web scraping and cover some of the ethical questions presented when scraping the Web. To begin, we’ll investigate browser-based web scraping: using a browser directly with Python to scrape content from the Web.

Browser-Based Parsing

Sometimes a site uses a lot of JavaScript or other post-page-load code to populate the pages with content. In these cases, it’s almost impossible to use a normal web scraper to analyze the site. What you’ll end up with is a very empty-looking page. You’ll have the same problem if you want to interact with pages (i.e., if you need to click on a button or enter some search text). In either situation, you’ll want to figure out how to screen read the page. Screen readers work by using a browser, opening the page, and reading and interacting with the page after it loads in the browser.

Tip

Screen readers are great for tasks performed by walking through a series of actions to get information. For this very reason, screen reader scripts are also an easy way to automate routine web tasks.

The most commonly used screen reading library in Python is Selenium. Selenium is a Java program used to open ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

AirBnbBlueOriginElectronic ArtsHomeDepotNasdaqRakutenTata Consultancy Services

QuotationMarkO’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
Julian F.
Head of Cybersecurity
QuotationMarkI wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.
Addison B.
Field Engineer
QuotationMarkI’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.
Amir M.
Data Platform Tech Lead
QuotationMarkI'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Mark W.
Embedded Software Engineer

You might also like

Data Wrangling with Python

Data Wrangling with Python

Dr. Tirthajyoti Sarkar, Shubhadeep Roychowdhury
Python for Data Analytics

Python for Data Analytics

O'Reilly Media, Inc.

Publisher Resources

ISBN: 9781491948804Errata Page