Hands-On Web Scraping with Python

Loading documents

In most cases, a document's content is obtained by using requests or urllib and is provided to pyquery as follows:

>>> from pyquery import PyQuery as pq>>> import requests>>> response = requests.get('http://www.example.com').text #content>>> from urllib.request import urlopen>>> response = urlopen('http://www.example.com').read()>>> docTree = pq(response)

pyquery can also load URLs using the Python library, urllib (default), or requests. It also supports requests-based parameters:

>>> pq("https://www.python.org")[<html.no-js>] >>> site=pq("https://www.python.org")>>> print(type(site))<class 'pyquery.pyquery.PyQuery'> >>> pq("https://www.samsclub.com")[<html>]

The pq object we obtained from the preceding code is being parsed ...