Python Web Scraping - Second Edition

Finding the owner of a website

For some websites it may matter to us who the owner is. For example, if the owner is known to block web crawlers then it would be wise to be more conservative in our download rate. To find who owns a website we can use the WHOIS protocol to see who is the registered owner of the domain name. A Python wrapper to this protocol, documented at https://pypi.python.org/pypi/python-whois, can be installed via pip:

   pip install python-whois

Here is the most informative part of the WHOIS response when querying the appspot.com domain with this module:

   >>> import whois   >>> print(whois.whois('appspot.com'))    {      ...      "name_servers": [        "NS1.GOOGLE.COM",         "NS2.GOOGLE.COM",         "NS3.GOOGLE.COM",         "NS4.GOOGLE.COM",  "ns4.google.com", ...