97 Things Every Data Engineer Should Know

Chapter 14. Consensual, Privacy-Aware Data Collection

Katharine Jarmul

How consensual is your data collection? Are customers aware of how their data is being used? Do they know how it is transformed, aggregated, fed into a model, and deployed? Is it being done with respect for their privacy? Can they ask for it not to be included in any of those automated processes? Do you have privacy-protection mechanisms to ensure that sensitive data is not at risk?

As data engineers, we aim for precision and efficiency. How can we get data from place A to place B, with the necessary in-between steps for cleaning, testing, and validation, and how can we do that most efficiently?

What gets lost in this objective to expedite and automate processing is that we are often handling personal data—sometimes extremely sensitive data. And if you’ve had the pleasure of working with European Union (EU) residents’ data, you might have thought about these topics further because of the General Data Protection Regulation (GDPR). How can we remove artifacts from users whose data we no longer have rights to? What does it mean for a person to be subject to automated processing? What does consensual data collection look like?

If we want to collect data consensually—for all people, not just those lucky enough to reside in the EU—we need to fundamentally change some of the ways we move data. Here are some ideas ...