Learn Python by Building Data Science Applications

Building our first task in Luigi

Luckily, luigi allows us to start small. We'll start by building a task that pulls all of the links on the battles, using the code from our wikiwwii package. First, we will import all we need in a separate file, luigi_fronts.py:

# luigi_fronts.pyfrom pathlib import Pathimport jsonimport luigifrom wikiwwii.collect.battles import collect_frontsURL = 'https://en.wikipedia.org/wiki/List_of_World_War_II_battles'folder = Path(__file__).parents[1] / 'data' 

Here, we declared a link for the battles, imported our collect_fronts function, and specified a relative folder to store the data in. Now, let's write the task itself. In the following, we'll create a task class, define the URL as a luigi parameter with a default ...