Let's now list all of the code that we'll optimize in future, based on the earlier description.
The first of the following points is quite simple: a single file script that takes care of scraping and saving in JSON format like we discussed earlier. The flow is simple, and the order is as follows:
The code is as follows:
from bs4 import BeautifulSoup import requests import json SO_URL = "http://scifi.stackexchange.com" QUESTION_LIST_URL = ...