
Temporally Aware Online News Mining and Visualization with Python 183
document we are currently working with. This is done by identifying the class name of the
div we would like to select using XPath notation. If you are comfortable with other selection
tools such as BeautifulSoup or lxml, you can use this instead of the Selector class offered
by Scrapy.
In this particular example, we know the name of the class that we would like to remove
from the document. Generally large companies such as Google AdWords or other advertise-
ment networks use a similar set of IDs for the divs where their advertisements are shown,
allowing easy identification and remov ...