97 Things Every SRE Should Know

Chapter 47. You Don’t Know for Sure Until It Runs in Production

Ingrid Epure

Netlify

The idea of testing in production is usually met with two types of reactions. Sometimes you get enthusiastic YAYs! Other times, you might get shocked, disapproving looks.

What explains this divide? This is one of the most interesting paradoxes of engineering. We often view production as this house of cards–like, fragile ecosystem that needs to be approached with care, silk gloves, or bunker gear. At the same time, we dream of observable systems and peaceful, pager-free nights.

Spoiler alert, you can’t have it both ways. So how do we bridge this gap?

For one, we must confront the myth that absolutely, under no circumstances, should bugs ever reach production. The idea of viewing shipped code as an experiment somehow implies that it’s not done or scrappy and that iterating while shipping is somehow bad.

You may feel that you can’t test in production because of a lack of good tooling. The bravest organizations attempt to build tools in-house, yet without a unified standard of running production experiments for the rest of us, that usually means scraping some Bash script together in the little time left between feature work sessions—or, even worse, making it an “ops problem.”

However, it’s far from an all-or-nothing approach; instead, it’s a sum of small things that can fundamentally change the developer ...