# Appendix C. SVM Dual Problem

To understand *duality*, you first need to understand the *Lagrange multipliers* method. The general idea is to transform a constrained optimization objective into an unconstrained one, by moving the constraints into the objective function. Let’s look at a simple example. Suppose you want to find the values of *x* and *y* that minimize the function *f*(*x*, *y*) = *x*^{2} + 2*y*, subject to an *equality constraint*: 3*x* + 2*y* + 1 = 0. Using the Lagrange multipliers method, we start by defining a new function called the *Lagrangian* (or *Lagrange function*): *g*(*x*, *y*, *α*) = *f*(*x*, *y*) – *α*(3*x* + 2*y* + 1). Each constraint (in this case just one) is subtracted from the original objective, multiplied by a new variable called a Lagrange multiplier.

Joseph-Louis Lagrange showed that if $(\hat{x},\hat{y})$ is a solution to the constrained optimization problem, then there must exist an $\hat{\alpha}$ such that $(\hat{x},\hat{y},\hat{\alpha})$ is a *stationary point* of the Lagrangian (a stationary point is a point where all partial derivatives are equal to zero). In other words, we can compute the partial derivatives of *g*(*x*, *y*, *α*) with regard to *x*, *y*, and *α*; we can find the points where these derivatives are all equal to zero; and the solutions to the constrained optimization problem (if they exist) must be among these stationary points.

In this example the partial derivatives are: $\left\{\begin{array}{l}\frac{\partial}{\partial x}g(x,y,\alpha )=2x-3\alpha \\ \frac{\partial}{\partial y}g(x,y,\alpha )=2-2\alpha \\ \frac{\partial}{\partial \alpha}g(x,y,\alpha )=-3x-2y-1\end{array}\right.$

When all these partial derivatives are equal to 0, we find that $2\widehat{x}-3\widehat{\alpha}=2-2\widehat{\alpha}=-3\widehat{x}-2$

Get *Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition* now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.