Hands-On APIs for AI and Data Science

Chapter 10. Using APIs in Data Pipelines

In their simplest form, pipelines may extract only data from one source such as a REST API and load to a destination such as a SQL table in a data warehouse. In practice, however, pipelines typically consist of multiple steps ... before delivering data to its final destination.

James Densmore Data Pipelines Pocket Reference (O’Reilly, 2021)

In Chapter 9, you used a Jupyter Notebook to query APIs and create data analytics. Querying directly in a notebook is useful for exploratory data analysis, but it requires you to keep querying the API over and over again. When data teams create analytics products for production, they implement scheduled processes to keep an up-to-date copy of source data in the format they need. These structured processes are called data pipelines because source data flows into the pipeline and is prepared and stored to create data products. Other common terms for these processes are Extract, Transform, Load (ETL) or Extract, Load, Transform (ELT), depending on the technical details of how they are implemented. Data engineer is the specialized role that focuses on the development and operation of data pipelines, but in many organizations, data scientists, data analysts, and infrastructure engineers also perform this work.

In this chapter, you will create a data pipeline to read SportsWorldCentral fantasy football player data using Apache Airflow, a popular open source tool for managing data pipelines using Python. ...