Over the past 10 years, the technology industry has experienced the emergence of “DevOps.” This new set of practices and tools have improved the velocity, quality, predictability, and scale of software engineering and deployment. Starting at the large Internet companies, the trend toward DevOps is now transforming the way that systems are developed and managed inside the enterprise—often dovetailing with enterprise cloud adoption initiatives. Regardless of your opinion about on-prem versus multitenant cloud infrastructure, the adoption of DevOps is improving how quickly new features and functions are delivered at scale for end users.
There is a lot to learn from the evolution of DevOps, across the modern Internet as well as within the modern enterprise—most notably for those who work with data every day.
At its core, DevOps is about the combination of software engineering, quality assurance, and technology operations (Figure 5-1). DevOps emerged because traditional systems management (as opposed to software development management) was not adequate to meet the needs of modern, web-based application development and deployment.
It’s time for data engineers and data scientists to embrace a new, similar discipline—let’s ...