A Practical Guide to Data Mining for Business and Industry

4 Data Preparation

Data Preparation

1. 4.1 Necessity of Data Preparation
2. 4.2 From Small and Long to Short and Wide
3. 4.3 Transformation of Variables
4. 4.4 Missing Data and Imputation Strategies
5. 4.5 Outliers
6. 4.6 Dealing with the Vagaries of Data
   1. 4.6.1 Distributions
   2. 4.6.2 Tests for Normality
   3. 4.6.3 Data with Totally Different Scales
7. 4.7 Adjusting the Data Distributions
   1. 4.7.1 Standardisation and Normalisation
   2. 4.7.2 Ranking
   3. 4.7.3 Box–Cox Transformation
8. 4.8 Binning
   1. 4.8.1 Bucket Method
   2. 4.8.2 Analytical Binning for Nominal Variables
   3. 4.8.3 Quantiles
   4. 4.8.4 Binning in Practice
9. 4.9 Timing Considerations
10. 4.10 Operational Issues

4.1 Necessity of Data Preparation

Having obtained useful data, it now needs to be prepared for analysis. It is not unusual to have the data stored at quite a detailed level in a data warehouse. But to get relevant, reliable and repeatable results out of the analyses, transformation and aggregation of the data is necessary. The type of aggregation has a major impact on the final result. It is unlikely that data mining algorithms will find hidden patterns without prior data preparation. Even if the user doing the data mining is not able to do the transformations and aggregations, it is important for the user to define the necessary steps and make sure someone else does them, for example, colleagues in the IT department. Most of the time, it is also useful to apply domain knowledge in the way the data preparation is done, for example, taking advantage of knowledge ...