Learning Python Networking - Second Edition
by José Manuel Ortega, Dr. M. O. Faruque Sarker, Sam Washington
Installation of Scrapy
There are diverse tools and techniques that allow a developer or analyst to access, consume, and extract content based on the web. The Scrapy project offers a tool that enables automated and rapid web scraping of large amounts of web-based content. Scrapy has very good documentation, which can be accessed from the following URL: https://doc.scrapy.org/en/latest.
Scrapy was created from Twisted (https://twistedmatrix.com/), so it is capable of performing thousands of queries simultaneously. Similarly, Scrapy makes use of tools such as BeautifulSoup and the Python XML package to facilitate content searches.
Scrapy needs lxml and OpenSSL as prerequisite packages for the installation. You can install Scrapy using pip with ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access