2
A Data Engineer’s Journey – Background Challenges
The purpose of this chapter is to explain the challenges you must face to navigate a successful journey as a data engineer. The intricate nature of managing vast data, evolving technologies, and ensuring efficient data pipelines are some of the hurdles that make data engineering a demanding field. In this chapter, we will explain why data engineering is hard, and provide a foundation for the development of principles to be discussed in later chapters. We will also provide insight when navigating these challenges. Finally, we will provide an overview of the data engineering approaches you can use to scope out the current and near-future technology landscape.
We will discuss three of the main ...
Get Data Engineering Best Practices now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
