Python Web Scraping Cookbook

How it works

The only difference in this recipe is how we fetch the resource:

req = urllib3.PoolManager()res = req.request('GET', url)

Unlike Requests, urllib3 doesn't apply header encoding automatically. The reason why the code snippet works in the preceding example is because BS4 handles encoding beautifully.  But you should keep in mind that encoding is an important part of scraping. If you decide to use your own framework or use other libraries, make sure encoding is well handled.