Errata

The last part of code snippet:
bsObj = BeautifulSoup(html)
downloadList = bsObj.findAll(src=True)
for download in downloadList:
fileUrl = getAbsoluteURL(baseUrl, download["src"])
if fileUrl is not None:
print(fileUrl)

urlretrieve(fileUrl, getDownloadPath(baseUrl, fileUrl, downloadDirectory))

should be
for download in downloadList:
fileUrl = getAbsoluteURL(baseUrl, download["src"])
if fileUrl is not None:
print(fileUrl)
urlretrieve(fileUrl, getDownloadPath(baseUrl, fileUrl,
downloadDirectory))

Note from the Author or Editor:
This was caused by an indentation error. It has been fixed in Github and will be fixed for future editions and prints of the book.

The code has serious bugs in handling internal Links. Here is a debugged code:

from urllib.request import urlopen
from urllib.error import HTTPError
from urllib.parse import urlparse
from bs4 import BeautifulSoup
import re
import random

#Retrieves a list of all Internal links found on a page
def getInternalLinks(bsObj, includeUrl):
internalLinks = []
#Finds all links that begin with a "/"
for link in bsObj.findAll("a",
href=re.compile("^(\/|.*(http:\/\/"+includeUrl+")).*")):
if link.attrs['href'] is not None and len(link.attrs['href']) != 0:
if link.attrs['href'] not in internalLinks:
internalLinks.append(link.attrs['href'])
return internalLinks

#Retrieves a list of all external links found on a page
def getExternalLinks(bsObj, url):
excludeUrl = getDomain(url)
externalLinks = []
#Finds all links that start with "http" or "www" that do
#not contain the current URL
for link in bsObj.findAll("a",
href=re.compile("^(http)((?!"+excludeUrl+").)*$")):
if link.attrs['href'] is not None:
if link.attrs['href'] not in externalLinks:
externalLinks.append(link.attrs['href'])
return externalLinks

def getDomain(address):
return urlparse(address).netloc

def followExternalOnly(bsObj, url):
externalLinks = getExternalLinks(bsObj, url)
if len(externalLinks) == 0:
print("Only internal links here. Try again.")
internalLinks = getInternalLinks(bsObj, getDomain(url))
if len(internalLinks) == 0:
return
if len(internalLinks) == 1:
randInternalLink = internalLinks[0]
else:
randInternalLink = internalLinks[random.randint(0, len(internalLinks)-1)]
if randInternalLink[0:4] != 'http':
randInternalLink = 'http://'+getDomain(url)+randInternalLink
if randInternalLink == url and len(internalLinks) == 1:
return
bsObjnext = BeautifulSoup(urlopen(randInternalLink), "html.parser")
#Try again
followExternalOnly(bsObjnext, randInternalLink)
else:
randomExternal = externalLinks[random.randint(0, len(externalLinks)-1)]
try:
nextBsObj = BeautifulSoup(urlopen(randomExternal), "html.parser")
print(randomExternal)
#Next page!
followExternalOnly(nextBsObj, randomExternal)
except HTTPError:
#Try again
print("Encountered error at "+randomExternal+"! Trying again")
followExternalOnly(bsObj, url)

url = "http://oreilly.com"
bsObj = BeautifulSoup(urlopen(url), "html.parser")
#Recursively follow external links
followExternalOnly(bsObj, url)

Note from the Author or Editor:
This code has been updated on Github and will be fixed in upcoming prints and editions of the book

The code says `name="image"`, but following page suggests (and code on actual site is) `name="uploadFile"`.

The link embedded in PDF for "That can be my next tweet!" is a wrong one, that should be
http://yes.thatcan.be/my/next/tweet/

Note from the Author or Editor:
The page has changed since the book was written. Updated for future editions

from urllib.request import urlopen
are appear twice – reduntdant.

Random external link is: http://igniteshow.com/
Random external link is: http://feeds.feedburner.com/oreilly/news
Random external link is: http://hire.jobvite.com/CompanyJobs/Careers.aspx?c=q319
Random external link is: http://makerfaire.com/
Should be read as
http://igniteshow.com/
http://feeds.feedburner.com/oreilly/news
http://hire.jobvite.com/CompanyJobs/Careers.aspx?c=q319
http://makerfaire.com/
Reflecting revised code print function, line 10 from the bottom of code snippet.

Note from the Author or Editor:
Updated code to reflect printout

#Finds all links that start with "http" or "www" that do
Should be read as
#Finds all links that start with "http" that do
To reflect the revised code line 8 from top

Note from the Author or Editor:
Changed the code to reflect this comment

insert
import json

this is missing in https://github.com/REMitchell/python-scraping/blob/master/chapter4/6-wikiHistories.py

Note from the Author or Editor:
The import statement has been added for future versions of the book

The paragraph and the 1st code refer/define main as exist,in the https://github.com/REMitchell/tesseract-trainer/blob/master/trainer.py code referred at the preceding paragraph.

However, there is no main method in this code example, instead you use __init__. So, the main should be read as __init__.

("['Software', 'Foundation']", 40), ("['Python', 'Software']", 38),....

should be read as
OrderedDict([("['Software', 'Foundation']", 40), ("['Python', 'Software']", 38),....

Note from the Author or Editor:
Updated to: "OrderedDict([('of the', 38), ('Software Foundation', 37), ..."

Google’s Geocode API,
should be read as
Google’s Geocoding API

from twitter import Twitter

shoulde be read as
from twitter import Twitter, OAuth

The last sentence tells:
This will create a new logfile, if one does not exist, in your current directory and output all logs and print statements to it.

this should be read as

This will create a new logfile, if one does not exist, in your current directory and output all logs to it.

The directory structure is different from the shown as:
• scrapy.cfg
— wikiSpider
— __init.py__
— items.py

This should be the following:
—scrapy.cfg
— wikiSpider
　— __init.py__
　— items.p

The linear rule number 4 at line 17 says;
"4. Optionally, write the letter "d" at the end." which does not say blank at the end,
however,
the line 22 regEx says
aa*bbbbb(cc)*(d | ), where blank comes at the end.
this should be read as the following to be consistent with the rule.
aa*bbbbb(cc)*(d|).

Note from the Author or Editor:
Changed text to different, more useful, example

Inside the getRandomExternalLink function in the if/else statement, the 'if' statement is set to return 'getNextExternalLink' if the length of externalLinks is equal to zero.

The 'getNextExternalLink' was never defined.

Note from the Author or Editor:
Updated code can be found in the github repository at: https://github.com/REMitchell/python-scraping/blob/master/chapter3/4-getExternalLinks.py

#second elif:
url = source[4:]
url = "http://"+source

#should be:
url = "http://"+source[4:]

In the Data Normalization section of chapter 7, there is a reference to recording the frequency of the 2-grams, then at the bottom of the page we are given a code snippet that introduces OrderedDict and uses the sorted function. In the sorted function the code contains ngrams.items() however the ngrams method returns a list and lists do not have an items() method. So the program generates an error.

In the next chapter, it looks like the code (at least on GitHub) has the ngrams function return a dictionary instead which allows the code in chapter 7 to work.

Note from the Author or Editor:
I mentioned the code that would accomplish this in passing, but did not actually include it. It will be included in future printings of the book, and in the ebook.

Traceback (most recent call last):
File "/home/dave/python/scrape_add_to_db.py", line 28, in <module>
links = getLinks("/wiki/Kevin_Bacon")
File "/home/dave/python/scrape_add_to_db.py", line 22, in getLinks
title = bsObj.find("h1").find("span").get_text()
AttributeError: 'NoneType' object has no attribute 'get_text'

I'm pretty sure that the error "None" means some problem downloading the url, but I know that I got pymysql working and changed my character sets. I thought that kindle might have mangled your nice code again so I went to github and copied and pasted the code, still same error. This is chapter 5 about 34% into the book (no page number on Kindle).

Note from the Author or Editor:
Unfortunately, Wikipedia has removed span tags from its titles, breaking some of the code in the book. This can be fixed by removing "find("span")" from the code, and just writing:
title = bsObj.find("h1").get_text()

This will be fixed in ebook editions and updated for future print editions.

.findAll("span", {"class": "green", "class": "red"})

an attempt to create a Python dict with repeated keys will preserve just the last one that is entered in the dict.

The correct would be:

.findAll("span", {"class": {"green", "red"}}

Note that we're passing now a collection (set) as the value for the "class" key on the attributes dict.

Note from the Author or Editor:
The line on page 16, in Chapter 2, should read:
.findAll("span", {"class":{"green", "red"}})