97 Things Every Data Engineer Should Know

Chapter 43. Let the Robots Enforce the Rules

Anthony Burdi

Dealing with messy inputs is part of our job as data professionals. We don’t always have to be pedantic, but asking data producers to structure their inputs in a certain way eases our burden with this not-so-glamorous work.

Great! Now our incoming data is cleaner. But then we realize that our burden just shifted to an interpersonal one: “asking” others to follow specific rules, read docs, and follow processes. For some of us, this is emotionally demanding and may end up making our jobs more difficult than if we just dealt with the messy data ourselves.

What do we do about it? The same thing we do with every other boring, repetitive, nitpicky, definable task: Pinky, automate it!

Here are a few validation-robot job description ideas:

• Use a Google form to capture data or visualization requests and simultaneously ask key questions—for example, what is the problem, why are you requesting it, what is the deliverable, the priority, etc. (See also Sam Bail’s wonderful blog post “Just One More Stratification!”.)

  Key bonus: Since the robot already asked your standard set of questions, you can start the conversation with the requester wielding some context, or even skip it altogether. Use a drop-down list (yay, validation!) rather than free-form text entry where possible.

  Double bonus: You can track the types of requests coming ...