book

Web Scraping with Python

by Ryan Mitchell

July 2015

Intermediate to advanced

256 pages

6h 28m

English

O'Reilly Media, Inc.

Read now

Unlock full access

What Is Web Scraping?Why Web Scraping?About This BookConventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
ConnectingAn Introduction to BeautifulSoupInstalling BeautifulSoupRunning BeautifulSoupConnecting Reliably
You Don’t Always Need a HammerAnother Serving of BeautifulSoupfind() and findAll() with BeautifulSoupOther BeautifulSoup ObjectsNavigating TreesRegular ExpressionsRegular Expressions and BeautifulSoupAccessing AttributesLambda ExpressionsBeyond BeautifulSoup
Traversing a Single DomainCrawling an Entire SiteCollecting Data Across an Entire SiteCrawling Across the InternetCrawling with Scrapy
How APIs WorkCommon ConventionsMethodsAuthenticationResponsesAPI CallsEcho NestA Few ExamplesTwitterGetting StartedA Few ExamplesGoogle APIsGetting StartedA Few ExamplesParsing JSONBringing It All Back HomeMore About APIs
Media FilesStoring Data to CSVMySQLInstalling MySQLSome Basic CommandsIntegrating with PythonDatabase Techniques and Good Practice“Six Degrees” in MySQLEmail
Document EncodingTextText Encoding and the Global InternetCSVReading CSV FilesPDFMicrosoft Word and .docx
Cleaning in CodeData NormalizationCleaning After the FactOpenRefine

Summarizing DataMarkov ModelsSix Degrees of Wikipedia: ConclusionNatural Language ToolkitInstallation and SetupStatistical Analysis with NLTKLexicographical Analysis with NLTKAdditional Resources
Python Requests LibrarySubmitting a Basic FormRadio Buttons, Checkboxes, and Other InputsSubmitting Files and ImagesHandling Logins and CookiesHTTP Basic Access AuthenticationOther Form Problems
A Brief Introduction to JavaScriptCommon JavaScript LibrariesAjax and Dynamic HTMLExecuting JavaScript in Python with SeleniumHandling RedirectsA Final Note on JavaScript
Overview of LibrariesPillowTesseractNumPyProcessing Well-Formatted TextScraping Text from Images on WebsitesReading CAPTCHAs and Training TesseractTraining TesseractRetrieving CAPTCHAs and Submitting Solutions
A Note on EthicsLooking Like a HumanAdjust Your HeadersHandling CookiesTiming Is EverythingCommon Form Security FeaturesHidden Input Field ValuesAvoiding HoneypotsThe Human Checklist
An Introduction to TestingWhat Are Unit Tests?Python unittestTesting WikipediaTesting with SeleniumInteracting with the SiteUnittest or Selenium?
Why Use Remote Servers?Avoiding IP Address BlockingPortability and ExtensibilityTorPySocksRemote HostingRunning from a Website Hosting AccountRunning from the CloudAdditional ResourcesMoving Forward
Installation and “Hello, World!”
Trademarks, Copyrights, Patents, Oh My!Copyright LawTrespass to ChattelsThe Computer Fraud and Abuse Actrobots.txt and Terms of ServiceThree Web ScraperseBay versus Bidder’s Edge and Trespass to ChattelsUnited States v. Auernheimer and The Computer Fraud and Abuse ActField v. Google: Copyright and robots.txt

Overview

Learn web scraping and crawling techniques to access unlimited data from any web source in any format. With this practical guide, you’ll learn how to use Python scripts and web APIs to gather and process data from thousands—or even millions—of web pages at once.

Ideal for programmers, security professionals, and web administrators familiar with Python, this book not only teaches basic web scraping mechanics, but also delves into more advanced topics, such as analyzing raw data or using scrapers for frontend website testing. Code samples are available to help you understand the concepts in practice.