Beautiful Testing

Conclusions

As Python has evolved from a small, obscure language to one used in production by companies all across the globe, we have striven to maintain a stable and reliable product despite huge growth in developers and language complexity. Without a strong suite of tests, Python never would have been able to manage this growth while maintaining its high bar for stability.

We were lucky that we adopted strong testing practices early on. It’s much easier to encourage newcomers to write comprehensive tests when they’re entering into a community in which it’s already a fundamental part of the culture. It’s just what we do here.

We believe that we’ve found a good balance that reinforces comprehensive testing practices without adding unnecessary hurdles that might discourage contributors, but this balance has evolved and changed as necessary throughout Python’s life cycle. We also now run a large number of tests in the background that contributors might not even be aware of. Our buildbots help us to catch problems on less common platforms, and continuous refleak checkers help identify those problems early. We run both static analysis tools (such as Coverity and Klocwork) and dynamic analysis tools (such as Valgrind and Fusil) for even more comprehensive coverage. Each and every one of these testing tools has proven its worth and caught problems that otherwise would have been put in front of real users. Despite these sophisticated tools, however, we still rely on basic unit tests; it ...