Data Engineering with Python

Chapter 3: Reading and Writing Files

In the previous chapter, we looked at how to install various tools, such as NiFi, Airflow, PostgreSQL, and Elasticsearch. In this chapter, you will be learning how to use these tools. One of the most basic tasks in data engineering is moving data from a text file to a database. In this chapter, you will read data from and write data to several different text-based formats, such as CSV and JSON.

In this chapter, we're going to cover the following main topics:

• Reading and writing files in Python
• Processing files in Airflow
• NiFi processors for handling files
• Reading and writing data to databases in Python
• Databases in Airflow
• Database processors in NiFi

Writing and reading files in Python

The title of this ...