Chapter 80. DataOps: DevOps for Data Management
Banjo Obayomi
As data becomes more ubiquitous in software projects, a culture has emerged around providing processes to manage data with a DevOps mindset. DataOps is a new paradigm that invokes DevOps principles and applies them to managing data to create value. As a cloud engineer, being able to enable your data team to work with the same efficiency as a software team is important, thanks to DevOps principles. Looking at problems with a DataOps lens will help ensure that your data team can deliver value to your end users. Here are three things to focus on when incorporating DataOps ideas into your processes.
Reproducible Data
As cloud engineers, we are used to spinning up and spinning down environments with a click of a button, but when it comes to collecting and storing data, the process is not always so clear-cut. Questions like “What if we can’t get data again?” or even worse, “We don’t know how we got our data” should scare any engineer. Our data pipelines should follow the same principles we use for tracking, deploying, and updating servers. This approach will help reduce the fear of not being able to get data again, and remove bottlenecks caused by barriers to sharing and collaborating with our data.
Analytics as Code
As cloud engineers, we have created numerous YAML files depicting ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access