Getting DataOps Right

Chapter 1. Introduction

Andy Palmer

Over the past three decades, as an enterprise CIO and a provider of third-party enterprise software, I’ve witnessed firsthand a long series of large-scale information technology transformations, including client/server, Web 1.0, Web 2.0, the cloud, and Big Data. One of the most important but underappreciated of these transformations is the astonishing emergence of DevOps.

DevOps—the ultimate pragmatic evolution of Agile methods—has enabled digital-native companies (Amazon, Google, etc.) to devour entire industries through rapid feature velocity and rapid pace of change, and is one of the key tools being used to realize Marc Andreessen’s portent that “Software Is Eating the World”. Traditional enterprises, intent on competing with digital-native internet companies, have already begun to adopt DevOps at scale. While running software and data engineering at the Novartis Institute of Biomedical Research, I introduced DevOps into the organization, and the impact was dramatic.

Fundamental changes such as the adoption of DevOps tend to be embraced by large enterprises after new technologies have matured to a point when the benefits are broadly understood, the cost and lock-in of legacy/incumbent enterprise vendors becomes insufferable, and core standards emerge through a critical mass of adoption. We are witnessing the beginning of another fundamental change in enterprise tech called “DataOps”—which will allow enterprises to rapidly and repeatedly ...