Going Pro in Data Science

Chapter 1. Introduction

Finding Signals in the Noise

Popular data science publications tend to creep me out. I’ll read case studies where I’m led by deduction from the data collected to a very cool insight. Each step is fully justified, the interpretation is clear—and yet the whole thing feels weird. My problem with these stories is that everything you need to know is known, or at least present in some form. The challenge is finding the analytical approach that will get you safely to a prediction. This works when all transactions happen digitally, like ecommerce, or when the world is simple enough to fully quantify, like some sports. But the world I know is a lot different. In my world, I spend a lot of time dealing with real people and the problems they are trying to solve. Missing information is common. The things I really want to know are outside my observable universe and, many times, the best I can hope for are weak signals.

CSC (Computer Sciences Corporation) is a global IT leader and every day we’re faced with the challenge of using IT to solve our customer’s business problems. I’m asked questions like: what are our client’s biggest problems, what solutions should we build, and what skills do we need? These questions are complicated and messy, but often there are answers. Getting to answers requires a strategy and, so far, I’ve done quite well with basic, simple heuristics. It’s natural to think that complex environments require complex strategies, but often they don’t. ...