Parallel coordinate descent

Definition

Parallel coordinate descent is a variant of gradient descent where we use a different learning rate in each coordinate. Explicitly, whereas with ordinary gradient descent, we define each iterate by subtracting a scalar multiple of the gradient vector from the previous iterate:

Ordinary gradient descent: ${\vec {x}}^{(k+1)}={\vec {x}}^{(k)}-\alpha ^{(k)}\nabla f\left({\vec {x}}^{(k)}\right)$

In parallel coordinate descent, we use a vector learning rate, i.e., we use a learning rate that could be different in each coordinate:

Parallel coordinate descent: for each coordinate $i$: ${\vec {x}}_{i}^{(k+1)}={\vec {x}}_{i}^{(k)}-\alpha _{i}^{(k)}\nabla f\left({\vec {x}}^{(k)}\right)$

Alternatively, using coordinate-wise vector multiplication, we can describe the above as:

${\vec {x}}^{(k+1)}={\vec {x}}^{(k)}-{\vec {\alpha }}^{(k)}*\nabla f\left({\vec {x}}^{(k)}\right)$