Practical Predictive Analytics

All subsets regression

All subsets regression is another method you can use for variable selection. It works by running separate regressions for different combinations of one variable, two variables, three variables at a time, until it has used all of the variables or until a specified stopping point. As part of the output, the best model for each of the combination of variables is then calculated. That will then give you an idea of which is the best single one variable model, two variable model, and so on, so that you can pare down a large list of variables into a smaller number of important ones using some plots and output statistics.