Data Science and Engineering at Enterprise Scale

Chapter 1. Sharing Information Across Disciplines in the Enterprise

Programs are meant to be read by humans and only incidentally for computers to execute.

Donald Knuth

This chapter will introduce you to the challenges of communicating ideas across multidisciplinary teams. While teams often have much in common in terms of skills and objectives, they may be composed of people from vastly different educational and cultural backgrounds, who bring different perspectives to bear on the same problem. In these environments, it is important to share information in a clear and consistent way. Notebooks provide an excellent way to do this, as they combine live code with formatted text so that programmers, data scientists, and even nontechnical members of the team can understand what is happening with various elements of the code being used.

The Overlap Between Data Scientist and Data Engineer

The modern data scientist on an enterprise team often has an intellectual ancestry in the academic world. The standard workflow in academic research is to measure something, compare the result to the predicted one, and report the findings in a peer-reviewed environment. The assumption in this environment is that “if you didn’t publish it, it didn’t happen,” which places a very heavy emphasis on careful documentation of work as the measure of success. It is not enough, however, to document and present your findings. As a data scientist, you must also be prepared to defend your position and persuade ...