Gradient descent with decaying learning rate

Definition

Gradient descent with decaying learning rate is a form of gradient descent where the learning rate varies as a function of the number of iterations, but is not otherwise dependent on the value of the vector at the stage. The update rule is as follows:

${\overrightarrow{x}}^{(k+1)}={\overrightarrow{x}}^{(k)}-{\alpha }_{k}f(\overrightarrow{x^{(k)}})$

where ${\alpha }_k$ depends only on $k$ and not on the choice of $x^{(k)}$.

Cases

Type of decay	Example expression for ${\alpha }_k$	More information
linear decay	${\alpha }_k=\frac{{\alpha }_0}{k+1}$	Gradient descent with linearly decaying learning rate
quadratic decay	${\alpha }_k=\frac{{\alpha }_0}{(k+1{)}^2}$	Gradient descent with quadratically decaying learning rate
exponential decay	${\alpha }_k={\alpha }_0{e}^{-\beta k}$ where $\beta >0$	Gradient descent with exponentially decaying learning rate