97 Things Every Data Engineer Should Know

Chapter 92. When to Avoid the Naive Approach

Nimrod Parasol

They always say you should avoid overengineering. You just started the project and have no way of knowing what the future holds, so you should make the solution as simple as possible. Don’t try to solve problems that do not exist yet. But sometimes the cost of postponing problems for later might be too high.

When you deal with large amounts of data, you find out that data migrations are complicated and costly processes. You need to write the code that transforms data from the old to the new format, then you need to run it over all your existing data, and eventually you need to perfectly synchronize the old and the new datasets in real time, so that consumers won’t notice any change. This process is expensive in employee power and computer power and always carries the risk of introducing bugs into your production system.

When you design your data store, you should think about how to avoid data migrations in the future. There are two main factors to consider.

The first is the data storage format. It might be a database, or the file format you choose to use in your data lake. Once you decide on one, changing it will require either having a complicated system that supports both the old and new formats, or doing a data migration. Both are unwanted.

The second aspect to consider is the partition’s schema. Most data stores today ...