Python Web Scraping Cookbook

There's more...

Requests and urllib3 are very similar in terms of capabilities. it is generally recommended to use Requests when it comes to making HTTP requests. The following code example illustrates a few advanced features:

import requests# builds on top of urllib3's connection pooling# session reuses the same TCP connection if # requests are made to the same host# see https://en.wikipedia.org/wiki/HTTP_persistent_connection for detailssession = requests.Session()# You may pass in custom cookier = session.get('http://httpbin.org/get', cookies={'my-cookie': 'browser'})print(r.text)# '{"cookies": {"my-cookie": "test cookie"}}'# Streaming is another nifty feature# From http://docs.python-requests.org/en/master/user/advanced/#streaming-requests ...