97 Things Every Data Engineer Should Know

Chapter 4. Analytics as the Secret Glue for Microservice Architectures

Elias Nema

Recently, we’ve seen a major shift toward microservice architectures. Driven by the industry’s most successful companies, this allowed teams to have fewer dependencies, move faster, and scale more easily. But, of course, it also introduced challenges. Most are related to the architecture’s distributed nature and the increased cost of communication.

Lots of progress has been made to overcome these challenges, mostly in the areas of system observability and operations. The journey itself is treated as a technical problem to solve. Analytics is often overlooked as something not having a direct relation to the system design. However, the heterogeneous nature of microservices makes a perfect case for data analysis.

That’s how data warehouses were born, after all—as central repositories of integrated data from one or more disparate sources. In a distributed setup, the role of the company-wide analytical platform can be immense. Let’s look at an example.

Imagine that your team releases a feature. You run an experiment and notice that the feature drives up your team’s target key performance indicator (KPI). That’s great. Should you roll it out for the entire user base? Sure, roll out, celebrate, and go home. What if, however, at the same time, another KPI for a different team goes down? This might happen ...