Working with cookies and sessions

In this section, we will be handling form processing for user authentication and managing cookies and sessions for http://quotes.toscrape.com/login from http://toscrape.com

In order to log in, you need to log in with a CSRF token (any username/password works).

Let's set up the code. The pyquery and requests libraries need to be imported and the required URLs will be collected and used. The getCustomHeaders() function, together with the cookieHeader argument, is used to set the cookie value for the URL request headers. The responseCookies() function, together with the response argument, displays the headers and cookies, and also returns the Set-Cookie value from cookies:

from pyquery import PyQuery as 

Get Hands-On Web Scraping with Python now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.