Applied Modeling Techniques and Data Analysis 1

4 Doubly Robust Data-driven Distributionally Robust Optimization

Data-driven distributionally robust optimization (DD-DRO) via optimal transport has been shown to encompass a wide range of popular machine learning algorithms. The distributional uncertainty size is often shown to correspond to the regularization parameter. The type of regularization (e.g. the norm used to regularize) corresponds to the shape of the distributional uncertainty. We propose a data-driven robust optimization methodology to inform the transportation cost underlying the definition of the distributional uncertainty. Empirically, we show that this additional layer of robustification, which produces a method we call doubly robust data-driven distributionally robust optimization (DD-R-DRO), allows the generalization properties of regularized estimators to be enhanced while reducing testing error relative to state-of-the-art classifiers in a wide range of datasets.

4.1. Introduction

A wide class of popular machine learning estimators have recently been shown to be particular cases of data-driven distributionally robust optimization (DD-DRO) formulations with a distributional uncertainty set centered around the empirical distribution (Abadeh et al. 2015; Blanchet et al. 2016, 2017; Gao and Kleywegt 2016).

For example, regularized logistic regression (Lee et al. 2006), support vector machines and sqrt-Lasso (Belloni et al. 2011), among many other machine learning formulations can be represented as DD-DRO ...