At some point in your career, you’ve probably been told that a given file should always be clean, and there’s no reason to spend time working on procedures to protect yourself from a problem. Inevitably, the impossible happens, and that previously perfect file has a problem, causing you to wake up at two o’clock in the morning to correct the production problem. If this has ever happened to you, or if you’re trying to not fall victim to this scenario, then this chapter is for you.
In the classic Problem-Design-Solution pattern, this chapter teaches you how to protect your package from breakages from future data problems, and provides new patterns for exploring every realm of data cleansing prior to loading or updating the data. Here’s what you can expect in this chapter: