Machine Learning

6.11 The Coordinate and Cyclic Coordinate Descent Methods

So far, we have discussed the steepest descent and Newton’s method for optimization. We will conclude the discussion with a third method, which can also be seen as a member of the steepest descent family of methods. Instead of the Euclidean and quadratic norms, let us consider the following minimization task for obtaining the normalized descent direction,

$\begin{array}{ll}\hfill \mathit{v}& =arg\underset{\mathit{z}}{min}{\mathit{z}}^{\text{T}}\nabla J,\hfill \end{array}$

  (6.57)

$\begin{array}{ll}\hfill \text{s.t.}& ||\mathit{z}|{|}_1=1,\hfill \end{array}$

  (6.58)

where ||⋅||₁ denotes the ℓ₁ norm, defined as

$||z|{|}_1:=\sum _{i=1}^l|z_i|.$

Most of Chapter 9 ...