97 Things Every Data Engineer Should Know

Chapter 93. When to Be Cautious About Sharing Data

Thomas Nield

When I was fresh out of business school, I was taught that silos are bad and even toxic to organizations. When people and departments become too secluded and inwardly focused, all kinds of corporate dysfunction can happen. Redundant work becomes commonplace, contradicting and competing objectives undermine the efficacy of the organization, and valuable data that could benefit everyone stays locked away with its owners.

While these concerns are valid and do happen, I have learned that bureaucracy and silos sometimes exist for legitimate reasons. Data can become a political and organizational headache if it is made available to too many parties. “Why?” you ask in a surprised tone. “Shouldn’t transparency and open data drive analysis and innovations?” Well, of course, but making data available is a balancing act and depends on the situation.

Let’s get the obvious reasons that you should not share data out of the way first. If the data is sensitive, it is best to give access on only a need-to-know basis. If the database is vulnerable to heavy traffic and expensive analytical SQL queries, that is another obvious reason to not provide access. Typically, in that situation, you would provide a replicated database used for analysis, so it does not impact the production database.

Now I want to get into a less obvious reason: ...