# Appendix C. SVM Dual Problem

To understand duality, you first need to understand the Lagrange multipliers method. The general idea is to transform a constrained optimization objective into an unconstrained one, by moving the constraints into the objective function. Let’s look at a simple example. Suppose you want to find the values of x and y that minimize the function f(x, y) = x2 + 2y, subject to an equality constraint: 3x + 2y + 1 = 0. Using the Lagrange multipliers method, we start by defining a new function called the Lagrangian (or Lagrange function): g(x, y, α) = f(x, y) – α(3x + 2y + 1). Each constraint (in this case just one) is subtracted from the original objective, multiplied by a new variable called a Lagrange multiplier.

Joseph-Louis Lagrange showed that if $\left(\stackrel{^}{x},\stackrel{^}{y}\right)$ is a solution to the constrained optimization problem, then there must exist an $\stackrel{^}{\alpha }$ such that $\left(\stackrel{^}{x},\stackrel{^}{y},\stackrel{^}{\alpha }\right)$ is a stationary point of the Lagrangian (a stationary point is a point where all partial derivatives are equal to zero). In other words, we can compute the partial derivatives of g(x, y, α) with regard to x, y, and α; we can find the points where these derivatives are all equal to zero; and the solutions to the constrained optimization problem (if they exist) must be among these stationary points.

In this example the partial derivatives are: $\left\{\begin{array}{l}\frac{\partial }{\partial x}g\left(x,y,\alpha \right)=2x-3\alpha \\ \frac{\partial }{\partial y}g\left(x,y,\alpha \right)=2-2\alpha \\ \frac{\partial }{\partial \alpha }g\left(x,y,\alpha \right)=-3x-2y-1\end{array}\right\$

When all these partial derivatives are equal to 0, we find that $2\stackrel{^}{x}-3\stackrel{^}{\alpha }=2-2\stackrel{^}{\alpha }=-3\stackrel{^}{x}-2$

Get Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition now with the O’Reilly learning platform.

O’Reilly members experience live online training, plus books, videos, and digital content from nearly 200 publishers.