Digital Signal Processing with Kernel Methods

5A Support Vector Machine Signal Estimation Framework

5.1 Introduction

SVMs were originally conceived as efficient methods for pattern recognition and classification (Vapnik, 1995), and the SVR was subsequently proposed as the SVM implementation for regression and function approximation (Shawe‐Taylor and Cristianini, 2004; Smola and Schölkopf, 2004). Nowadays, the SVR and other kernel‐based regression methods have become a mature and recognized tool in DSP. This is not incidental, as the widespread adoption of SVM by researchers and practitioners in DSP is a direct consequence of their good performance in terms of accuracy, sparsity, and flexibility.

Early studies of time series with supervised SVM algorithms paid attention mainly to two DSP signal models; namely, nonlinear system identification and time series prediction (Drezet and Harrison 1998; Goethals et al. 2005b; Gretton et al. 2001b; Mattera 2005; Pérez‐Cruz and Bousquet 2004; Suykens 2001; Suykens et al. 2001b). However, the algorithm used in both of them was the conventional SVR, just working on time‐lagged samples of the available time series (i.e., essentially an ad hoc time embedding). Although good results have been reported with this approach, several concerns can be raised from a conceptual viewpoint of estimation theory:

1. The basic assumption for the regression problem statement, in an MMSE sense, is i.i.d. observations. This assumption is not at all fulfilled in time‐series analysis, and algorithms neglecting ...