Learn the art of efficient web scraping and crawling with Python
About This Book
- Extract data from any source to perform real time analytics.
- Full of techniques and examples to help you crawl websites and extract data within hours.
- A hands-on guide to web scraping and crawling with real-life problems and solutions
Who This Book Is For
If you are a software developer, data scientist, NLP or machine-learning enthusiast or just need to migrate your company's wiki from a legacy platform, then this book is for you. It is perfect for someone , who needs instant access to large amounts of semi-structured data effortlessly.
What You Will Learn
- Understand HTML pages and write XPath to extract the data you need
- Write Scrapy spiders with simple Python and do web crawls
- Push your data into any database, search engine or analytics system
- Configure your spider to download files, images and use proxies
- Create efficient pipelines that shape data in precisely the form you want
- Use Twisted Asynchronous API to process hundreds of items concurrently
- Make your crawler super-fast by learning how to tune Scrapy's performance
- Perform large scale distributed crawls with scrapyd and scrapinghub
This book covers the long awaited Scrapy v 1.0 that empowers you to extract useful data from virtually any source with very little effort. It starts off by explaining the fundamentals of Scrapy framework, followed by a thorough description of how to extract data from any source, clean it up, shape it as per your requirement using Python and 3rd party APIs. Next you will be familiarised with the process of storing the scrapped data in databases as well as search engines and performing real time analytics on them with Spark Streaming. By the end of this book, you will perfect the art of scarping data for your applications with ease
Style and approach
It is a hands on guide, with first few chapters written as a tutorial, aiming to motivate you and get you started quickly. As the book progresses, more advanced features are explained with real world examples that can be reffered while developing your own web applications.
Table of contents
- Table of Contents
- Learning Scrapy
- About the Author
- About the Reviewer
1. Introducing Scrapy
- Hello Scrapy
- More reasons to love Scrapy
- About this book: aim and usage
- The importance of mastering automated data scraping
- Being a good citizen in a world full of spiders
- What Scrapy is not
- 2. Understanding HTML and XPath
3. Basic Crawling
- Installing Scrapy
- UR2IM – the fundamental scraping process
- A Scrapy project
- Extracting more URLs
4. From Scrapy to a Mobile App
- Choosing a mobile application framework
- Creating a database and a collection
- Populating the database with Scrapy
- Creating a mobile application
- 5. Quick Spider Recipes
- 6. Deploying to Scrapinghub
7. Configuration and Management
- Using Scrapy settings
- Stopping crawls early
- HTTP caching and working offline
- Crawling style
- Downloading media
- Amazon Web Services
- Using proxies and crawlers
- Further settings
- 8. Programming Scrapy
9. Pipeline Recipes
- Using REST APIs
- Interfacing databases with standard Python clients
- Interfacing services using Twisted-specific clients
- Interfacing CPU-intensive, blocking, or legacy functionality
10. Understanding Scrapy's Performance
- Scrapy's engine – an intuitive approach
- Getting component utilization using telnet
- Our benchmark system
- The standard performance model
- Solving performance problems
- Troubleshooting flow
11. Distributed Crawling with Scrapyd and Real-Time Analytics
- How does the title of a property affect the price?
- Overview of our distributed system
- Changes to our spider and middleware
- Creating our custom monitoring command
- Calculating the shift with Apache Spark streaming
- Running a distributed crawl
- System performance
- The key take-away
A. Installing and troubleshooting prerequisite software
- Installing prerequisites
- The system
- Installation in a nutshell
- Installing on Linux
- Installing on Windows or Mac
System setup and operations FAQ
- What do I download and how much time does it take?
- What should I do if Vagrant freezes?
- How do I shut down/resume the VM quickly?
- How do I fully reset the VM?
- How do I resize the virtual machine?
- How do I resolve any port conflicts?
- How do I make it work behind a corporate proxy?
- How do I connect with the Docker provider VM?
- How much CPU/memory does each server use?
- How can I see the size of Docker container images?
- How can I reset the system if Vagrant doesn't respond?
- There's a problem I can't work around, what can I do?
- Title: Learning Scrapy
- Release date: January 2016
- Publisher(s): Packt Publishing
- ISBN: 9781784399788
You might also like
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition
Through a series of recent breakthroughs, deep learning has boosted the entire field of machine learning. …
40 Algorithms Every Programmer Should Know
Learn algorithms for solving classic computer science problems with this concise guide covering everything from fundamental …
51+ hours of video instruction. Overview The professional programmer’s Deitel® video guide to Python development with …
Tiny Python Projects
The projects are tiny, but the rewards are big: each chapter in Tiny Python Projects challenges …