Learn Python by Building Data Science Applications

Writing to SQL

In many cases, it is preferable or more convenient to write to a database rather than a flat-file. Let's illustrate this case with our 311 data pipeline.

Writing data to a database is quite similar and we won't need to change much. One major difference is task completion detection—for an obvious reason, there is no file to check for existence. As a workaround, luigi creates a utility table that stores unique records of the complete tasks. This process is integral to the framework, so most of the time, there is no reason for us to think about it. With that being said, SQL-based pipelines have two, pretty strong, caveats:

• As the task does not result in an isolated task, there is no simple way to pull data from this specific ...