Website Scraping with Python: Using BeautifulSoup and Scrapy

© Gábor László Hajba 2018

Gábor László HajbaWebsite Scraping with Pythonhttps://doi.org/10.1007/978-1-4842-3925-4_5

5. Handling JavaScript

Gábor László Hajba¹ 

(1)

Sopron, Hungary

 

This chapter is all about handling websites that utilize JavaScript to render information dynamically.

You have seen in the previous chapters that a basic website scraper loads the web page’s contents and does its extraction on this source code. And if there’s JavaScript included, it’s not executed, and the dynamic information is missing from the page.

This is bad, at least in those cases where you need that dynamic data.

Another interesting part of scraping websites that use JavaScript is that you may need clicks or button presses to go to the right page / get the right content, ...