book

Data Ingestion with Python Cookbook

Name: Data Ingestion with Python Cookbook
Author: Gláucia Esppenchutz
ISBN: 9781837632602

by Gláucia Esppenchutz

May 2023

Intermediate to advanced

414 pages

8h 39m

English

Packt Publishing

Read now

Unlock full access

Data Ingestion with Python Cookbook
ContributorsAbout the authorAbout the reviewers
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedSectionsGetting readyHow to do it…How it works…There’s more…See alsoGet in touchShare Your ThoughtsDownload a free PDF copy of this book
Part 1: Fundamentals of Data Ingestion
Chapter 1: Introduction to Data Ingestion
Technical requirementsSetting up Python and its environmentGetting readyHow to do it…How it works…There’s more…See alsoInstalling PySparkGetting readyHow to do it…How it works…There’s more…See alsoConfiguring Docker for MongoDBGetting readyHow to do it…How it works…There’s more…See alsoConfiguring Docker for AirflowGetting readyHow to do it…How it works…See alsoCreating schemasGetting readyHow to do it…How it works…See alsoApplying data governance in ingestionGetting readyHow to do it…How it works…See alsoImplementing data replicationGetting readyHow to do it…How it works…There’s more…Further reading
Chapter 2: Principals of Data Access – Accessing Your Data
Technical requirementsImplementing governance in a data access workflowGetting readyHow to do it…How it works…See alsoAccessing databases and data warehousesGetting readyHow to do it…How it works…There’s more…See alsoAccessing SSH File Transfer Protocol (SFTP) ﬁlesGetting readyHow to do it…How it works…There’s more…See alsoRetrieving data using API authenticationGetting readyHow to do it…How it works…There’s more…See alsoManaging encrypted ﬁlesGetting readyHow to do it…How it works…There’s more…See alsoAccessing data from AWS using S3Getting readyHow to do it…How it works…There’s more…See alsoAccessing data from GCP using Cloud StorageGetting readyHow to do it…How it works…There’s more…Further reading
Chapter 3: Data Discovery – Understanding Our Data before Ingesting It
Technical requirementsDocumenting the data discovery processGetting readyHow to do it…How it works…Configuring OpenMetadataGetting readyHow to do it…How it works…There’s more…See alsoConnecting OpenMetadata to our databaseGetting readyHow to do it…How it works…Further readingOther tools
Chapter 4: Reading CSV and JSON Files and Solving Problems
Technical requirementsReading a CSV ﬁleGetting readyHow to do it…How it works…There’s more…See alsoReading a JSON ﬁleGetting readyHow to do it…How it works…There’s more…See alsoCreating a SparkSession for PySparkGetting readyHow to do it…How it works…There’s more…See alsoUsing PySpark to read CSV ﬁlesGetting readyHow to do it…How it works…There’s more…See alsoUsing PySpark to read JSON ﬁlesGetting readyHow to do it…How it works…There’s more…See alsoFurther reading
Chapter 5: Ingesting Data from Structured and Unstructured Databases
Technical requirementsConfiguring a JDBC connectionGetting readyHow to do it…How it works…There’s more…See alsoIngesting data from a JDBC database using SQLGetting readyHow to do it…How it works…There’s more…See alsoConnecting to a NoSQL database (MongoDB)Getting readyHow to do it…How it works…There’s more…See alsoCreating our NoSQL table in MongoDBGetting readyHow to do it…How it works…There’s more…See alsoIngesting data from MongoDB using PySparkGetting readyHow to do it…How it works…There’s more…See alsoFurther reading
Chapter 6: Using PySpark with Deﬁned and Non-Deﬁned Schemas
Technical requirementsApplying schemas to data ingestionGetting readyHow to do it…How it works…There’s more…See alsoImporting structured data using a well-deﬁned schemaGetting readyHow to do it…How it works…There’s more…See alsoImporting unstructured data without a schemaGetting ready…How to do it…How it works…Ingesting unstructured data with a well-deﬁned schema and formatGetting readyHow to do it…How it works…There’s more…See alsoInserting formatted SparkSession logs to facilitate your workGetting readyHow to do it…How it works…There’s more…See alsoFurther reading
Chapter 7: Ingesting Analytical Data
Technical requirementsIngesting Parquet ﬁlesGetting readyHow to do it…How it works…There’s more…See alsoIngesting Avro filesGetting readyHow to do it…How it works…There’s more…See alsoApplying schemas to analytical dataGetting readyHow to do it…How it works…There’s more…See alsoFiltering data and handling common issuesGetting readyHow to do it…How it works…There’s more…See alsoIngesting partitioned dataGetting readyHow to do it…How it works…There’s more…See alsoApplying reverse ETLGetting readyHow to do it…How it works…There’s more…See alsoSelecting analytical data for reverse ETLGetting readyHow to do it…How it works…See alsoFurther reading

Part 2: Structuring the Ingestion Pipeline
Chapter 8: Designing Monitored Data Workﬂows
Technical requirementsInserting logsGetting readyHow to do it…How it works…See alsoUsing log-level typesGetting readyHow to do it…How it works…There’s more…See alsoCreating standardized logsGetting readyHow to do it…How it works…There’s more…See alsoMonitoring our data ingest ﬁle sizeGetting readyHow to do it…How it works…There’s more…See alsoLogging based on dataGetting readyHow to do it…How it works…There’s more…See alsoRetrieving SparkSession metricsGetting readyHow to do it…How it works…There’s more…See alsoFurther reading
Chapter 9: Putting Everything Together with Airﬂow
Technical requirementsInstalling AirflowConfiguring AirflowGetting readyHow to do it…How it works…See alsoCreating DAGsGetting readyHow to do it…How it works…There's more…See alsoCreating custom operatorsGetting readyHow to do it…How it works…There's more…See alsoConﬁguring sensorsGetting readyHow to do it…How it works…See alsoCreating connectors in AirﬂowGetting readyHow to do it…How it works…There's more…See alsoCreating parallel ingest tasksGetting readyHow to do it…How it works…There's more…See alsoDeﬁning ingest-dependent DAGsGetting readyHow to do it…How it works…There's more…See alsoFurther reading
Chapter 10: Logging and Monitoring Your Data Ingest in Airﬂow
Technical requirementsInstalling and running AirflowCreating basic logs in AirﬂowGetting readyHow to do it…How it works…See alsoStoring log files in a remote locationGetting readyHow to do it…How it works…See alsoConfiguring logs in airflow.cfgGetting readyHow to do it…How it works…There’s more…See alsoDesigning advanced monitoringGetting readyHow to do it…How it works…There’s more…See alsoUsing notiﬁcation operatorsGetting readyHow to do it…How it works…There’s more…Using SQL operators for data qualityGetting readyHow to do it…How it works…There’s more…See alsoFurther reading
Chapter 11: Automating Your Data Ingestion Pipelines
Technical requirementsInstalling and running AirflowScheduling daily ingestionsGetting readyHow to do it…How it works…There's more…See alsoScheduling historical data ingestionGetting readyHow to do it…How it works…There's more…Scheduling data replicationGetting readyHow to do it…How it works…There's more…Setting up the schedule_interval parameterGetting readyHow to do it…How it works…See alsoSolving scheduling errorsGetting readyHow to do it…How it works…There’s more…Further reading
Chapter 12: Using Data Observability for Debugging, Error Handling, and Preventing Downtime
Technical requirementsDocker imagesSetting up StatsD for monitoringGetting readyHow to do it…How it works…See alsoSetting up Prometheus for storing metricsGetting readyHow to do it…How it works…There’s more…Setting up Grafana for monitoringGetting readyHow to do it…How it works…There’s more…Creating an observability dashboardGetting readyHow to do it…How it works…There’s more…Setting custom alerts or notiﬁcationsGetting readyHow to do it…How it works…Further reading
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Content preview from Data Ingestion with Python Cookbook

9 Putting Everything Together with Airﬂow

So far, we have covered the different aspects and steps of data ingestion. We have seen how to configure and ingest structured and unstructured data, what analytical data is, and how to improve logs for more insightful monitoring and error handling. Now is the time to group all this information to create something similar to a real-world project.

From now on, in the following chapters, we will use Apache Airflow, an open source platform that allows us to create, schedule, and monitor workflows. Let’s start our journey by configuring and understanding the fundamental concepts of Apache Airflow and how powerful this tool is.

In this chapter, you will learn about the following topics:

Configuring Airflow ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Time Series Analysis with Python Cookbook

Publisher Resources

ISBN: 9781837632602

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Data Ingestion with Python Cookbook

by Gláucia Esppenchutz

9

Putting Everything Together with Airﬂow

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.