Hands-On APIs for AI and Data Science

Chapter 10. Using APIs in Data Pipelines

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 10th chapter of the final book. The GitHub repo for this book is https://github.com/handsonapibook/apibook-part-one.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

In their simplest form, pipelines may extract only data from one source such as a REST API and load to a destination such as a SQL table in a data warehouse. In practice, however, pipelines typically consist of multiple steps ... before delivering data to its final destination.

James Densmore, Data Pipelines Pocket Reference, (O’Reilly, 2021)

In Chapter 9, you used a Jupyter Notebook to query APIs and create data analytics. Querying directly in a Notebook is useful for exploratory data analysis, but it requires you to keep querying the API over and over again. When data teams create analytics products for production, they implement scheduled processes to keep an up-to-date copy of source data in the format they need. These structured processes are called data pipelines because source data flows into the pipeline and is prepared and stored to create data products. ...