Python Web Scraping - Second Edition

Executing JavaScript

To confirm your WebKit installation can execute JavaScript, there is a simple example available at http://example.webscraping.com/dynamic.

This web page simply uses JavaScript to write Hello World to a div element. Here is the source code:

<html>     <body>         <div id="result"></div>         <script>         document.getElementById("result").innerText = 'Hello World';         </script>     </body> </html> 

With the traditional approach of downloading the original HTML and parsing the result, the div element will be empty, as follows:

>>> import lxml.html>>> from chp3.downloader import Downloader>>> D = Downloader()>>> url = 'http://example.webscraping.com/dynamic' >>> html = D(url) >>> tree = lxml.html.fromstring(html) >>> tree.cssselect('#result')[0].text_content() ...