Data Engineering with Scala and Spark

8

Test-Driven Development, Code Health, and Maintainability

In this chapter, we are going to look at some software development best practices and learn how to apply them to data engineering. The topics covered in the chapter will go a long way to helping you identify defects early, write code in a consistent way, and address potential security vulnerabilities as part of development. For example, test-driven development (TDD) requires building test cases first before we write the actual application code. Since we start with test cases, this helps to write the application code in a way that can be easily used to run the test cases. Another example is code formatting. Each of us has our own ways of writing programs and having consistency among ...