Python Web Scraping Cookbook

How to do it

We proceed with the recipe as follows:

1. Execute the file as a Python script:

$ python elasticcloud_starwars.py

2. This will loop through up to 20 characters and drop them into the sw index with a document type of people. The code is straightforward (replace the URL with yours):

from elasticsearch import Elasticsearchimport requestsimport jsonif __name__ == '__main__':    es = Elasticsearch(        [            "https://elastic:[e-mail]:9243"        ])i = 1while i<20:    r = requests.get('http://swapi.co/api/people/' + str(i))    if r.status_code is not 200:        print("Got a " + str(r.status_code) + " so stopping")        break j = json.loads(r.content) print(i, j) #es.index(index='sw', doc_type='people', ...