97 Things Every Data Engineer Should Know

Chapter 36. Give Data Products a Frontend with Latent Documentation

Emily Riederer

The rise of DataOps demonstrates the value of bringing DevOps principles to data engineering. Similar, but less explored, is the need to incorporate principles from design and product management into data builds and explicitly craft a good user experience.

Engineers should think critically about building a “frontend” for their data that makes it easy to understand and intuitive to use. For data, specifically, the frontend is not a traditional UI but rather the lump sum of tools and documents that enable users to understand the intent, lineage, and quality of a dataset.

Of course, building this frontend is no small task, and often data engineers are at full capacity with the technical aspects of their work. However, many of the artifacts that data consumers want can be created with little to no incremental effort if engineers embrace latent documentation: systematically documenting their own thought processes and decision making during the engineering process in a way that can be easily shared with and interpreted by users.

The following are examples of latent documentation you can collect for a low-cost data frontend:

Generate a data dictionary while gathering user requirements

When you talk to users about data requirements (you are talking to users, aren’t you?), engage them in the process of ...